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TASK  LEVEL  JOB  PERFORMANCE  CRITERIA  DEVELOPMENT 


1.  INTRODUCTION 


This  is  an  account  of  the  major  steps  and  findings  in  the  execution  of  Contract  F4 1609-7 l-C-OO  10, 
secured  by  the  Occupational  Research  Division  of  the  Air  Force  Human  Resources  Laboratory  (AFHRLX 
Brooks  Air  Force  BasCj  Texas,  and  performed  by  the  American  Institutes  for  Research  (AIR),  Washington, 
DC,  during  the  period  1971  through  1974.  The  basic  document  for  a large  portion  of  this  account  is  the 
final  management  report,  having  the  same  title,  authored  by  tlie  Principal  Investigator,  Mr*  Qifford  P, 
Hahn.  Extensive  extracts  have  been  made  from  that  report  and  condensed  for  presentation  here*  Tlie 
contract  report  and  its  appendices  are  available  from  the  Defense  Documentation  Center  (Hahn,  1975), 

Perhaps  no  better  explanation  of  the  aims  of  the  study  can  be  had  than  to  quote  the  Introduction  and 
Background  section  of  the  contract  proposal  of  16  April  1970. 

One  of  the  greatest  needs  of  managers  of  the  military  manpower  and  personnel  systems  is  for  a 
method  to  accurately  measure  how  well  individuals  perform  on  the  job*  The  Personnel  Research 
Division  (AFHRL)  has  developed  techniques  by  which  the  Air  lorce  can  deter  mine  the  tasks  and  jobs 
being  assigned  to  person neL  but  little  or  no  information  is  available  concerning  how  well  these  tasks 
and  jobs  arc  being  performed.  Official  supervisory  ratings  do  not  serve  this  purpose  well  enough.  Such 
ratings  are  global  in  nature,  not  specitieally  related  to  tasks  and  jobs,  highly  inflated,  and  provide 
insufficient  variance  for  discriminating  among  individuals  being  rated. 

The  upsl^ot  is  that  selection  and  classification  devices  are  designed  to  maximLe  performance  in 
school,  rather  than  performance  on  the  job.  Training  courses  are  presumably  tailored  to  job  content, 
but  adequate  procedures  are  not  available  to  determine  their  efficiency.  Prollciency  tests  contain 
questions  about  tasks  likely  to  be  encountered,  but  are  not  validated  against  job  performance.  In 
short,  there  is  no  way  to  demonstrate  that  individuals  with  high  aptitude  scores,  who  have  undergone 
extensive  formal  training,  and  w'ho  score  high  on  proficiency  tests,  actually  perform  signillcantly 
better  on  the  job  than  individuals  having  lower  aptitude  scores  and  less  training,  and  scoring  lower  on 
proficiency  tests.  Until  better  criteria  are  available,  it  will  be  difficult  to  evaluate  new  selection 
devices,  training  techniques,  occupational  structures,  assignment  procedures,  classification  models,  or  a 
host  of  other  management  pmgrams  and  devices* 

There  appears  to  be  little  hope  of  obtaining  good  performance  ratings  from  supervisors  as  long  as 
such  ratings  are  (a)  revealed  to  the  ratee  and  (b)  used  for  making  decisions  about  the  ratee^s 
promotions*  Nor  is  it  feasible  to  develop  objective  performance  tests  by  which  subordinates  can  be 
evaluated  by  unbiased  observers  in  a controlled  setting*  Tiere  are  approximately  30,000  tasks 
performed  by  Air  Force  enlisted  personnel.  It  would  not  be  feasible  to  construct  and  administer 
performance  tests  except  in  a tew  critical  areas.  11  is  also  recognized  that  how  an  individual  behaves  for 
a short  period  of  time,  when  he  knows  he  is  being  closely  observed,  docs  not  necessarily  correlate  with 
how  well  he  behaves  in  the  operational  environment,  when  he  is  not  being  evaluated  closely*  Thus*  one 
is  throw' n back  to  obtaining  performance  information  from  those  who  arc  in  a position  to  observe 
workers  in  an  operational  setting. 

The  foregoing  is  sufficient  to  sketch  the  magnitude  of  the  problem  and  to  identify  the  many 
applications  that  the  data  might  have  if  they  were  available*  It  is  not  a problem  that  has  been  peculiar  to  the 
Air  Force,  and  it  was  viewed  with  such  concern  that  initial  funding  of  the  contract  was  provided  by  the 
Department  of  Defense* 

The  task  rating  approach  to  job  performance  was  designed  to  provide  information  that  cannot  be  had 
when  performance  is  assessed  on  a global  basis*  It  was  known  that  jobs  differ  greatly  with  respect  to  the 
tasks  comprising  them,  althougli  the  Air  Force  specialty  codes  (AFSC)held  by  the  incumbents  in  those  jobs 
might  be  identical*  If  it  were  possible  to  reduce  performance  measurements  to  elements  (tasks)  of  jobs 
reliably  and  with  agreement  among  judges,  the  relative  importance  of  many  factors  contributing  to  overall 
performance  might  be  revealed.  In  particular,  it  might  be  possible  to  weigli  the  contribution  of  an 
incumbenfs  aptitude,  experience,  and  attitude  toward  establislung  an  overall  judgment  regarding  his 
performance.  If  both  the  task  ratings  and  the  overall  ratings  were  made  by  the  same  judges,  control  wu)uld 
be  exercised  over  sources  of  error.  It  would  not  be  necessary  to  make  suppositions  regarding  the 
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equivalence  of  task  performance  raters  in  one  sample  and  overall  performance  raters  in  another.  If  two  or 
more  specialties  were  studied,  it  would  be  possible  to  compare  them  with  respect  to  die  way  performance 
factors  assembled  themselves  to  yield  overall  assessments. 


iK  BACKGROUND 

A number  of  ground  rules  were  set  forth  in  the  work  statement  of  the  contract,  most  of  which  could 
be  followed  to  the  letter,  but  some  of  which  proved  to  be  infeasible  in  the  operational  situation.  An  outline 
of  the  full  sequence  of  contract  and  analysis  events  is  given  as  Figure  1. 


‘ Agency 

Activity 

AFHRL 

Prepares  statement  of  vwork  and  awards  competitive  bid  to  AIR. 

AIR 

Procures  information  on  AFSCs  from  files  and  rtarrows  specialty  list. 

AIR 

Begins  base  interviews  with  NCOs  to  select  specialties* 

AIR 

Continues  base  visits  and  prepares  materials  for  field  reviews.  Narrows  list  to  three  specialties,  selects 
coordinators,  and  prepares  materials  for  assembled  meetir)9s  with  50  NCOs  per  specialty. 

AIR 

Hosts  a week-long  meeting  with  coordinators  for  each  AFSC  at  Lackland  AFB,  Texas.  Tries  out 
preliminary  scales  after  reviewing  selected  task  items  and  performance  dimensions.  Sends  materials  to 
printer. 

AFHRL 

Selects  the  experimental  test  battery,  with  AIR  inputs  to  items*  Provides  base  rosters  for  surveys. 

AIR 

Mails  out  survey  materials  to  base  coordinators  via  Consolidated  Base  Personnel  Offices.  Produces  testing 
manual. 

AFHRL 

Procures  Survey  Control  Number  arwl  prints  test  battery.  Sends  test  batteries  through  Test  Control 
Officers.  Scores  returned  tests  and  prepares  card  image  tape  of  test  data  for  AIR. 

AIR 

Records  survey  responses  and  makes  initial  report  on  number  of  cases  collected. 

AFHRL 

Decides  to  perform  supplementary  survey  and  selects  materials  and  survey  procedures,  using  normal 
channels-  Provides  additional  case  rosters,  and  has  returned  surveys  directed  to  AIR. 

AFHRL 

Obtains  address  list  of  personnel  surveyed  at  Time  1 and  still  in  the  Air  Force  at  Time  2.  Obtains  Time  2 
ratings  for  direct  delivery  to  AIR.  Scores  supplemental  survey  tests  and  sends  card  image  tape  to  AIR, 

AIR 

Delivers  original  survey  data  tape  file  to  AFHRL. 

AFHRL 

Runs  shakedown  tests  on  AIR  data  tape.  Performs  studies  of  interrater  reliabilities  paralleling  AIR 
analyses*  Classifies  verbal  comments  made  in  peer  and  supervisor  reports. 

AIR 

Performs  correlations  among  and  overall  ratings^  studies  aptitude/sktils  vs*  motivation  ratings. 

AFHRL  8e  AIR 

Hold  coordination  meeting  on  final  analyses,  deciding  to  eliminate  additional  analyses  of  incumbents  as 
performance  raters,  and  to  limit  computed  means  to  two  raters.  Establish  a limited  set  of  data  of  record 
predictors,  covering  all  essential  demographic  variables. 

AIR 

Delivers  zero-order  correlations  of  variables  and  first  portion  of  final  report.  Contract  ended  31 
December  1974. 

AFHRL 

Delivers  to  AIR  tape  file  of  predictor  variables  after  matching  cases  with  Air  Force  data  record  files. 
Runs  regression  analyses  to  see  if  suitably  scored  attitude  and  satisfaction  tests  could  predict  both  tad< 
and  overall  performance  ratings.  Determine  overall  rating  reliability  with  cross-rater  regressions.  Charts 
future  analyses  by  using  graphical  interpretations. 

AIR 

Delivers  data  ar>d  correlation  printouts  with  additional  final  analyses,  ending  final  report.  Delivers  Time  2 
data  tapes. 

AFHRL 

Continues  regression  analyses,  setting  up  the  "flawed  sample'^  procedure.  Analyzes  relative  contributions 
of  task  ratings  and  data  of  record.  Selects  techniques  to  reduce  the  number  of  predictors,  and  tests 
contributions  of  aptitude  and  demographic  data  against  just  5 task  performance  ratings  as  predictors. 
Runs  analyses  to  determine  if  rater  tendency  was  cause  of  findings.  Sets  up  files  and  begins  analyses  to 
answer  questions  not  covered  by  the  report*  Writes  the  report. 

Figure  L Sequence  of  the  contract  effort. 
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Considerable  effort  was  expended  upon  selection  of  the  specialties  to  be  used  in  the  study.  The  aid  of 
noncommissioned  officer  (NCO)  consultants  was  obtained  from  Major  Commands,  and  visits  lasting  up  to  a 
week  were  made  to  bases*  Tliis  ended  with  the  choice  of  three  AFSCs:  29150,  Telecommunications 
Operations  Specialist;  30454,  Ground  Radio  Communications  Equipment  Repainuan;  and  43151C,  Aircraft 
Maintenance  Specialist  (single-  and  dual -engine  jet).  The  choice  of  specialties  fulfilled  a number  of  contract 
stipulations  aimed  at  collecting  data  that  would  represent  the  Air  Force  broadly*  Among  the  conditions 
required  were  adequate  numbers  of  job  incumbents,  presence  of  both  hardware  and  software  specialty 
activities,  availability  of  bases  for  direct  visitation,  availability  of  a current  job  inventory  in  the  specialty, 
and  at  least  one  specialty  that  included  personnel  who  did  not  receive  formal  technical  school  training.  It 
chanced  that  all  three  chosen  specialties  operated  in  shifts*  Only  die  communications  center  sample 
contained  women,  practically  all  of  whom  were  telephone  switchboard  operators* 

Selection  of  the  specialties  was  accomplied  by,  and  followed  by,  development  of  draft  form  rating 
instruments  and  the  choice  of  an  initial  set  of  tasks  for  tryout.  Approximately  50  NCQs  in  each  AFSC  were 
convened  by  the  contractor  for  a week’s  experimental  conference  and  workshop  that  were  held  successively 
at  the  Lackland  AFB,  Texas,  facility  of  AFHRL  At  this  time,  the  NCOs  were  indoctrinated  in  performance 
evaluation  and  observation.  Then  they  used  the  instruments  and  discussed  the  observability  of  the  selected 
tasks.  Problems  and  facts  pertinent  to  task  performance  ratings  were  brought  out  in  each  specialty,  resulting 
in  separate  rating  dimensions  for  certain  tasks.  The  usefulness  of  the  ratings  for  future  Air  Force  decisions 
was  emphasized,  and  the  NCQs  were  returned  to  their  bases  to  act  as  focal  points  for  indoctrinating 
additional  raters,  and  to  coordinate  the  surveys  to  follow.  It  was  here  that  difficulties  arose.  The  survey 
control  number  (SCN)  system  was  introduced  into  the  Air  Force  while  the  survey  forms  were  being  printed, 
and  these  emerged  unnumbered.  No  surveys  without  a control  number  were  authorized,  and  the  1972 
surveys  could  not  be  authorized  by  base  personnel  offices  without  a control  number.  During  the  resulting 
delays  some  of  the  chosen  NCOs  were  transferred  and  participant  interest  in  the  study  waned*  Data  returns 
were  slow,  unpredictable,  often  incomplete,  and  smaller  tlian  anticipated* 

The  job  incumbent  was  asked  to  complete  a standard  Air  Force  job  inventory,  a booklet  in  which  he 
rated  himself  on  the  performance  of  those  tasks  he  did  in  the  selected  set,  a booklet  in  which  he  indicated 
his  source  of  knowledge  in  or  experience  at  each  task  he  performed,  and  another  booklet  indicating  the 
type  of  satisfaction  or  annoyance  he  had  from  performing  each  task*  The  incumbent  also  rated  10  job 
factors  covering  his  whole  job.  No  exact  check  was  made  to  determine  how  many  of  these  incumbents  also 
acted  as  peer  raters. 

Since  any  specific  task  was  performed  by  less  than  half  the  individuals  in  most  samples,  and  some 
tasks  were  performed  by  very  few  incumbents^  it  was  essential  to  have  fairly  large  groups  in  order  to  have 
stable  data  at  the  task  level.  There  was,  for  example,  no  way  of  assuring  that  two  raters  would  rate  an 
incumbent  on  the  same  tasks,  although  the  probability  of  their  doing  so  was  high*  As  the  difficulties  of  data 
collection  developed,  it  was  feared  that  the  major  objectives  of  the  contract  would  not  be  met  without 
additional  sampling.  In  die  fall  of  1973,  AFHRL  supplemented  the  AIR  data  by  sending  out  additional 
surveys  and  a battery  of  selected  tests.  These  were  handled  throu^  the  test  control  officers,  and  the  data 
recovery  improved.  In  the  supplemental  survey,  incumbents  did  not  fill  out  job  inventories  nor  the  work 
factor  ratings;  but  they  were  asked  to  complete  the  three  survey  forms  and  to  take  the  test  battery.  Besides 
the  incumbent,  only  one  rater,  a supervisor,  was  used. 

Time  2 surveying  of  previously  rated  incumbents  began  in  early  1974*  All  incumbents  available  fora 
year  were  located,  and  their  bases  were  requested  to  have  them  rated  by  two  supervisors,  wherever  possible* 
Incumbents  did  not  participate  and  were  not  informed. 

In  the  spring  of  1974,  the  complete  data  tapes  of  the  Time  1 surveys  were  forwarded  by  the 
contractor  to  AFHRL  and  analyses  were  begun  at  both  places.  AFHRL  pei formed  both  parallel  and 
independent  analyses  to  investigate  features  of  the  data  of  future  interest,  as  well  as  those  of  immediate 
concern.  In  October  1974,  an  analysis  planning  conference  was  held  to  determine  which  analyses  were 
essential  for  the  contractor  to  perform  with  existing  funds.  (It  had  never  been  planned  to  exhaust  the  data 
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reservoir  to  provide  the  contract  final  report,  since  it  was  evident  that  many  tangential  problems  would 
prove  to  be  interesting,)  The  present  report  extends  data  analysis  activities  beyond  those  of  the  contractor 
in  order  to  achieve  closure  on  findings  regarded  as  having  major  interest  for  the  Air  Force,  Some  additional 
work  that  was  performed  subsequently  by  AFHRL  is  reported  elsewhere  (Wiley,  1976). 


Ill*  DEVELOPMENT  OF  PERFORMANCE  MEASUREMENT 
INSTRUMENTS  AND  PROCEDURES 

Tliis  section  is  quoted  in  full  from  the  contractor’s  management  report,  pages  6 through  9 (Hahn, 
1975). 

A major  task  in  developing  the  criterion  instruments  and  procedures  sought  for  field  use 
involved  transiaring  the  descriptive  task  statements  of  the  USAF  Job  Inventories  into  evaluative 
statements  that  could  form  the  basis  for  scaling  how  well  the  tasks  were  done.  Throughout  the  entire 
development  process,  contractor  staff  numbers  who  were  expcriencxid  in  the  techniques  of 
performance  evaluation  and  scaling  had  the  active  support  of  a group  of  experienced  incumbents  from 
the  career  ladders  being  studied.  This  was  accomplished  through  an  interative  scries  of  working 
sessions,  first  with  groups  of  2 to  3 incumbents  per  ladder,  then  groups  of  10,  and  finally  groups  of 
approximately  50  incumbents  for  each  career  ladder. 

The  Job  Inventory  task  lists  were  reviewed  and  appropriate  additions,  selections,  and  revisions 
were  made  in  order  to  update  the  task  lists.  Data  from  iterative  .sorts  of  discernible  levels  of 
performance;  degree  of  observability  and  measurability;  criticality : range  of  difficulty;  and  stability  of 
performance  were  used  to  develop  an  initial  Ust  of  candidate  tasks  for  further  development.  Career 
ladder  incumbents  then  developed  an  initial  set  of  behavioral  descriptors  for  total  task  performance  or 
for  critical  dimensions  of  performance  for  the  tasks  on  the  candidate  list. 

A series  of  one^veek  work  sessions  with  groups  of  10  experienced  incumbents  were  conducted 
to  review  and  revise  the  descriptors  previously  developed,  to  develop  additional  descriptors  for 
relevant  tasks,  and  to  devise  initial  scaling  procedures  for  use  with  the  descriptors.  Tliese  same  groups 
rated  the  importance  of  each  task  dimension  for  inclusion  in  the  field  survey  forms.  The  data  from 
these  sessions  were  used  by  the  contractor  staff  in  preparing  forms  for  use  in  simulated  rating  sessions 
by  larger  groups  of  NCOs  from  each  career  ladder* 

A scries  of  week-long  ladder  workshops  was  held,  each  involving  approximately  50  senior  NCOs 
from  the  career  fields  being  studied.  The  purposes  of  these  workshops  were  to  use  previously 
developed  instruments  in  a simulated  rating  situation,  to  develop  additional  scales,  to  develop  ancillary 
instruments  for  field  use  in  conjunction  with  the  performance  rating  instruments,  and  to  elicit 
opinions  about  procedures  to  be  used  to  coUect  the  field  data. 

Results  from  the  career  ladder  workshops  indicated  that  the  forms  developed  appeared  capable 
of  capturing  some  of  the  performance  variance  that  existed  in  the  field.  Six  overall  job  performance 
ratings  were  generated  in  addition  to  the  ratings  at  the  task  level,  as  were  judgments  of  importance  for 
inclusion  of  the  surviving  tasks  in  the  final  field  format.  Data  from  the  workshops  were  utilized  to 
make  final  revisions  in  the  field  forms  for  collecting  performance  data  and  these  were  later  reviewed 
by  workshop  participants  on  a mail<iut  basis. 

Several  additional  forms  were  developed  and  tried  out  in  the  career  ladder  workshops.  These 
forms  were  to  be  utilized  in  connection  with  the  performance  data  survey  to  satisfy  other  concerns  of 
interest  to  the  monitoring  agency.  One  of  these  was  a Work  Factor  Requirements  Rating  Form.  The 
work  factors  concerned  were  those  which  applied  to  the  job  itself  and  not  to  the  airmen  performing 
the  job  nor  the  manner  in  which  it  was  performed.  These  were  the  types  of  factors  that  are  typically 
considered  during  job  evaluation  proceed urcs  designed  to  establish  an  appropriate  grade  and  pay  level. 

It  was  anticipated  that  such  factors  would  eventually  be  compared  with  performance  data.  A 
ten-factor  form  with  a nine-point  scale  for  rating  each  factor  was  developed  for  field  use. 

In  order  to  understand  better  some  of  the  factors  which  contribute  to  task  performance, 
instruments  and  procedures  were  also  developed  to  collect  data  regarding  the  acquisitioi^  and  retention 
of  the  skills  and  knowledges  associated  with  various  tasks.  Tire  form  developed  called  for  Judgments  of 
the  major  source  of  skill  acquisition  in  terms  of  technical  training  school,  a formal  OJT  program,  or 
job  experience.  Separate  ratmgs  were  requested  for  acquisition  of  job  knowledge  and  job  proficiency 
following  the  model  of  the  Air  Force  dual-channel  OJT  concept.  Judgments  were  also  requested 
concerning  the  relative  perisli ability  of  task  knowledge  and  proficiency  after  an  acceptable  level  had 
once  been  attained. 

Tentative  forms  were  also  developed  forjudging  the  interest  value  and  judged  complexity  of  the 
various  tasks.  These  ratings  tended  to  be  highly  intercorrelated  and  their  relationship  to  overall 
performance  was  somewhat  unstable  across  the  three  career  ladders.  Data  regarding  these  task 
characteristics  were  therefore  not  sought  from  the  field. 
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Tentative  forms  were  also  developed  for  obtaining  task  preference  ratings  on  the  premise  that 
individual  preferences  for  certain  tasks  or  groups  of  tasks  might  affect  the  motivation  level  and  thus 
influence  task  performance*  l or  tasks  selected  as  most  and  least  preferable,  judgments  of  the  relative 
poteney  of  generalized  motivational  factors  were  requested.  The  data  from  these  activities  were  used 
to  prepare  a motivation  rating  form  to  be  completed  by  incumbents.  This  form  allowed  for 
expressions  of  both  the  importance  of  the  motivational  factors  and  the  direction  of  their  influence; 
le.,  positive,  negative,  or  both* 

Data  Sources 

As  a result  of  the  developmental  activities  described,  the  foEowing  set  of  survey  instruments  was 
utilized  for  coUecting  data  for  5-skill -level  incumbents  from  the  three  career  ladders. 

L Performance  and  Skills j Abilities  Versus  Motivation  Ratings.  This  rating  instrument  was  designed 
for  use  by  both  supervisors  and  pee  is  to  provide  data  on  the  level  of  task  dimension  performance  and  on 
the  relative  importance  of  skills  and  abilities  as  opposed  to  motivational  factors  in  contributing  to  the  level 
of  performance. 

2.  Performance  Ratings,  This  rating  instrument  was  designed  for  use  by  incumbents  to  provide  data 
concerning  their  own  perceived  level  of  task  dimension  performance. 

3.  Motivation  Ratings,  This  rating  instrument  was  designed  for  use  by  incumbents  to  provide  data 
regarding  both  the  intensity  and  direction  of  effects  of  a set  of  generalized  motivational  factors. 

4*  TYaining  and  Skill  Retention  Ratings,  This  rating  instrument  was  designed  for  use  by  incumbents 
well  as  by  supervisors  and  peers  to  provide  data  regarding  the  primary  source  for  acquisition  of  task 
knowledge  and  task  proficiency,  as  well  as  the  relative  perishability  of  such  knowledge  and  proficiency  after 
an  acceptable  level  had  once  been  attained. 

5.  Work  Requirement  Factor  Ratings,  This  rating  instrument  was  designed  for  use  by  incumbents, 
supervisors,  and  peers  to  provide  data  on  ten  generalized  requirement  factors  associated  with  the  duty 
positions  within  tlie  career  ladder  themselves  rather  than  with  incumbents  or  their  level  of  performance. 

5.  United  States  Air  Force  Job  Inventory.  Copies  of  the  current  job  inventory  were  reproduced  and 
used  by  incumbents  for  indicating  which  tasks  they  performed  and  the  relative  amount  of  time  spent  on 
each. 


IV,  THE  SURVEY  MATERIAIS 

General  instructions  aimed  at  motivating  respondents  were  provided  to  all  participants  in  the  surveys, 
and  each  booklet  contained  a specific  set  of  instructions.  Correlative  data  were  obtained  by  matching 
incumbent  names  and  social  security  numbers  against  Air  Force  personnel  record  tapes  mmntained  by 
AFHRL  While  the  matched  data  were  not  part  of  the  survey,  much  of  tlie  matched  data  figured 
prominently  in  the  analyses.  In  addition  to  the  general  orientation  provided  to  all  participants,  peers  and 
supervisors  were  given  tips  about  observing  and  rating  job  performance. 

Besides  completing  job  inventories,  incumbents  rated  themselves  on  task  performance  in  the  first 
survey  booklet,  reported  their  reactions  to  performing  the  same  tasks  in  the  second  booklet,  and  in  the 
third  booklet  gave  information  regarding  their  sources  of  training  and  their  retention  of  skills.  They 
indirectly  appraised  the  requirements  of  their  jobs  by  rating  work  requirement  factors. 

Peers  and  supervisors  rated  specific  incumbents  on  task  performance,  by  task,  and  using  a booklet 
identical  to  the  incumbenf  s,  rated  the  tasks  on  sources  of  training  and  skill  preservation. 

Those  incumbents  in  the  initial  survey  who  complied  with  all  the  requested  activities  completed  the 
following: 

1.  A current  Air  Force  job  inventory  in  their  ladder 

2.  A self  rating  booklet  of  task  performance  dimensions  (7-point  scale) 
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3*  A niotivation-supplied-by-task  rating  booklet 

4.  A task  training  source  and  skill  retention  booklet 

5-  An  experimental  lest  battery,  containing,  11  short  co^nitve  tests,  a biographical  inventory,  a 
400-iiem  Vocational  Interest -Career  Examination  (VOICE),  a leas t-preferred-co worker  set  of  scaled  items, 
and,  a 43-iteni  list  of  job  satisfaction  determiners  to  be  rated 

6.  A single  rating  page  of  10  work  requirement  factors. 

Peer  and  Supervisor  Perfonnance  Rating  Booklet 

Tlie  peer  and  supervisor  perfonnance  rating  booklet  came  next,  with  the  overall  rating  page,  which  is 
part  of  the  same  booklet.  It  was  preceded  by  information  concerning  tips  about  observing  and  rating  job 
performance. 

The  peer  and  supervisor  performance  rating  booklet  contained  information  identifying  the  ratee,  his 
job  title,  grade,  skill  level,  the  length  of  time  he  was  known  by  the  rater,  estimates  of  the  amount  and  kind 
of  contact  the  rater  had  with  the  ratee,  and  provided  space  for  general  comments. 

The  Experimentai  Test  Battery 

The  experimental  test  battery  was  administered  by  AFHRL.  Task  scores  for  these  cognitive  tests  are 
included  in  Table  1.  Also  included  in  the  battery  were  tlie  Biographical  Inventory  and  VOICE,  the 
Vocational  Interest*  Career  Examination.  These  were  followed  by  the  Least  Preferred  Coworker  rating 
instrument  and  a 43'item  Job  Satisfaction  Information  List.  (Tire  biographical  material  would  have  to  be 
updated  extensively  for  current  use,  and  the  VOICE  instrument  is  in  the  process  of  refinement  by  AFHRL.) 

Some  Specific  Comments  on  the  Suri^ey  Materials 

A single  item  of  the  43-item  job  satisfaction  list,  the  one  concerned  with  how  the  Air  Force  meets  its 
commitments  to  the  individual,  was  of  strictly  temporary  interest.  Ttiis  item  reflected  attitudes  existing 
toward  Air  Force  service  in  1972,  rather  than  attitudes  toward  the  incumbent’s  specific  job.  Graphical 
analyses  showed  that  the  majority  of  the  responses  were  unfavorable,  and  this  was  true  for  all  three  AFSCs. 
The  responses  were  the  most  extremely  negative  of  all  43  items  in  the  list. 

Other  attitude  and  interest  items  have  been  grouped  for  coding  and  weighting  purposes,  then 
examined  graphically.  Interesting  comparisons  among  the  three  AFSCs  resulted.  To  cite  but  one  of  these, 
the  graphs  showed  that  aircraft  mechanics  frequently  considered  their  working  conditions  to  be  poor,  but 
that  their  general  satisfaction  with  their  job  assignments  was  predominantly  better  than  that  shown  by 
personnel  in  the  other  two  AFSCs. 

The  instructions  for  the  incumbent  task  motivation  booklet  may  have  presented  the  incumbent  with 
a hard  problem,  if  he  was  conscientious  in  his  responses.  Since  it  is  impossible  to  know  whether  or  not  the 
incumbent  understood  these  complicated  instructions,  or  was  responding  in  any  but  a perfunctory  manner, 
it  is  doubtful  that  definitive  analyses  can  be  made  of  the  motivation  booklet  responses. 

Analyses  will  be  presented  with  only  part  of  the  incumbent’s  possible  contribution  - his  self-ratings 
on  task  performance,  and  bis  actual  perfonnance  on  tests.  Tire  analysis  of  peer  and  supervisory  data  will  be 
primarily  concerned  with  performance  ratings,  and  they  cannot  utilize  information  about  job  relations  with 
the  ratee. 


V.  SAMPLE  SIZES 

Rather  than  list  all  the  types  of  data  combinations,  certain  classes  of  data  have  been  analyzed  for  the 
largest  possible  N.  When  intercorrelations  are  being  computed,  the  matrices  must  be  complete,  and  the  data 
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Table  L Selected  Samples,  the  ""Flagged  Population” 


Identifying  Variable 

AFSC  2i1X0 

N = 457 

AFSC  a04X4 

N = 399 

AFSC  431  XIC 

N = 497 

Mean 

SD 

Mean 

SO 

Mean 

SD 

Demographic  Data  of  Record 

Grade  (1-9) 

4.118 

.648 

4.150 

.706 

4.528 

.558 

Montlis,  tot  act  Fed  mil  service 

55.821 

46.084 

52.331 

37.630 

58.883 

21.973 

Sex  male  = 1 / other  = 0 

398' 

.335 

399' 

.000 

487' 

.000 

Age  in  months  at  enlistment 

292.068 

47,764 

290.088 

38.884 

293.655 

26.156 

Educational  level  at  enlist. 

4.112 

,457 

4,135 

.550 

4.019 

.422 

Size  of  city  of  origin  (1-5) 

2.626 

1.316 

2.609 

1.287 

2.448 

1.280 

Married  “ 1 / other  = 0 

257' 

.496 

233' 

.493 

355' 

.445 

Divorced  or  separated  = 1/0 

3' 

,081 

4' 

.100 

9' 

.135 

Single  = 1 / other  = 0 

197' 

.495 

162' 

.491 

122' 

.433 

Aptitude  Data  of  Record,  Aptitude  Indexes 

Mechanical  AI 

47.269 

21.558 

77.043 

15.070 

63.737 

13.039 

Administrative  AI 

71.543 

13.443 

75.627 

15.535 

51.306 

19.257 

General  AI 

63.260 

17.011 

78.922 

13.691 

55,934 

17.016 

Electronic  AI 

58.107 

19.027 

85,952 

9.569 

58.809 

17.365 

Experimental  Cognitive  Test  Scores 

Decoding 

19.050 

12.448 

25.501 

11.058 

18.875 

11.840 

Memory  for  landmarks 

16.042 

9.309 

20.378 

8.769 

14.694 

8.548 

Complex  scale  reading 

4.125 

3.099 

6.652 

4.363 

4.780 

3.545 

Pursuit 

29.234 

10.705 

32.343 

10.588 

29.869 

12.361 

Figure  analogies 

20.449 

7.968 

26.566 

5.605 

19.975 

8,450 

Hands 

28.630 

17.554 

36.153 

13.075 

33.187 

14.610 

Cubes 

14.230 

6.514 

19.333 

6.425 

16,039 

6.528 

Mechanical  principles 

34.184 

15.073 

57.384 

18.479 

44.522 

17.493 

Following  directions 

25.179 

18,964 

34.947 

18.528 

23.528 

18.442 

Practical  estimations 

7.619 

3,738 

9.797 

4,003 

8.290 

3.701 

Spatial  reasoning 

5.212 

8.648 

1 2.466 

10.605 

3.943 

8.416 

Overall  Performance  Ratings  By  Peers  and/or  Supervisors  Combined 

General  performance 

82.065 

13.177 

76.426 

15.357 

80.491 

13.589 

Amount  of  work  performed 

82.565 

14.406 

75.265 

16.590 

80.360 

14.595 

Quality  of  work 

83.875 

14.154 

79.220 

15.523 

83.686 

13.515 

Will  do  more  than  his  share 

81.374 

17.636 

74.184 

20.487 

77.644 

19.423 

Self- initiating 

83.435 

16.291 

75.044 

20.196 

79.767 

18.735 

Will  share  knowledge 

84.082 

15.013 

80.015 

16.484 

82.923 

16.181 

^Converted  to  actual  n* 


Note.  “ The  educational  scale  used  in  Tabic  1 is  as  follows: 

1 = elementary  school,  graduated  or  not 

2 = high  school,  one  through  three  years 

3 = completed  high  school  equivalency  tests  but  has  no  diploma 

4 = high  school  graduate 

5 = one  or  more  years  of  college,  includes  an  AA  degree,  or  a graduate  of  a diploma  school  program,  including 

registered  nurse,  but  not  a 4-ycar  college  diploma 

6 ^ one  or  more  bacbelor^s  degrees,  includes  optometry  and  podiatry 

7 = master's  candidate  under  US  A FIT 

B = master degree  and  above  in  anything  less  than  a doctorate 
9 = all  earned  doctor's  degrees,  also  LLB  and  JD 


iKivc  been  stripped  down  for  all  cells  to  the  same  types  of  cases  within  each  AFSC,  In  the  one  instance  of 
task  performance  ratings  used  in  regression  problems,  special  matrices  had  to  be  constructed  to  provide  the 
same  number  of  entries  in  all  cells.  Tlie  minimum  N for  any  sample  providing  all  types  of  data  used  in  tliesc 
analyses  appears  for  those  regression  problems  in  which  complete  data  were  demanded  tor  overall  ratings, 
test  battery  scores,  and  personnel  data.  Tlie  three  Ns  in  tliat  instance  were  457  for  AFSC  291X0,  399  for 
304  X4,  and  487  for  431X1C.  (These  samples  ap(>ear  in  Table  1.)  Subsamplc  Ns  for  task  performance 
ratings  range  from  the  very  small  to  the  hundreds. 


VI.  DESCRIPTION  OE  THREE  SAMPLES 

To  describe  the  population  of  this  study,  one  is  faced  with  the  alternative  of  supplying  tlie  means  and 
sigmas  for  the  largest  sample  measured  on  each  variable  or  of  selecting  a group  of  incumbents  whose 
membership  contains  only  individuals  with  data  on  every  variable.  Tlie  latter  choice  was  made  alter 
comparing  means  and  standard  deviations  of  larger  populations  to  determine  it  the  sample  statistics  were 
Tcprescntativc  of  the  population  at  large.  Table  1 gives  the  means  and  standard  deviations  of  the  three 
incumbent  samples  used  in  regression  problems  to  determine  the  relative  contribution  ol  various  kinds  ot 
measures  toward  accounting  for  the  six  overall  pc rfo nuance  ratings. 

Tlic  tabic  meajis  and  standard  deviations  supply  striking  facts  about  incumbents  in  AFSC  304X4:  (a) 
they  were  slightly  younger  chronolo^cally  and  in  service  time  than  members  of  the  other  two  groups,  (b) 
they  entered  the  Air  Force  slightly  better  educated  than  the  others,  (c)  they  excelled  the  other  two  groups 
on  all  Aptitude  indexes,  and  (d)  they  excelled  the  other  two  groups  on  every  cognitive  test:  but,  tlicir  six 
performance  ratings  were,  dimertsion-by'diniension,  lower  than  those  of  the  other  two  groups. 

The  only  significant  characierislic  of  the  291X0  AFSC  sample  shown  in  the  table  ^s  that  59  ol  the 
457  incumbents  were  female,  the  only  women  in  the  study.  In  general,  the  291X0  AFSC  performance 
ratings  were  the  highest.  Tlic  performance  ratings  ol  the  AFSC  431X1C  sample  showed  the  greatest 
variability  among  the  performance  rating  dimensions.  Compared  to  the  data  revealing  the  high  aptitude  and 
lower  performance  rating  ol  incumbents  in  AFSC  304X4  these  items  are  trivial. 

Variables  relating  to  time  on  base  ajid  job,  which  were  employed  by  the  contractor,  were  not 
available  for  participants  in  the  supplemental  survey,  and  were  thus  not  available  lor  the  three  selected 
samples  in  all  cases.  Task  rating  data  arc  represettted  sporadically  througlvout  the  study  and  cannot  be 
provided  in  a summary  table. 


Vn.  ANALYSES  OF  UNRESTRICTED  SAMPLES 
Distributions  of  Task  Performance  Ratings 

Tlie  contractor  has  made  extensive  analyses  of  the  use  of  the  T-point  scales  ol  task  performance  by 
[ncumbents,  peers,  and  supervisors.  These  appear  in  tabular  lonii  in  the  management  report  (Ualin,  1975). 
The  following  critical  questions  were  raised: 

1 . Was  there  systematic  variance  of  any  sort  in  task  perfonnance  ratings? 

2.  Were  some  tasks  systematically  rated  as  better  perfonned  than  others? 

3.  Were  there  systematic  differences  between  incumbent  sell-ratings,  peer  ratings  of  incumbents,  and 
supervisor  ratings  of  incumbents? 

4.  Were  there  systematic  differences  among  the  three  AFSCs  in  ilic  rated  perlo nuance  levels  of 

tasks? 

floth  the  contractor  and  AFHRL  ran  distributions  which  produced  nearly  identical  results  during  the 
shakedown  phase  of  the  analyses.  Tlie  contractor  tables  liave  been  used  and  have  been  condensed  to  provide 
Figures  2 and  3.  Figure  2 was  derived  from  AIR  data  converted  to  the  percentages  of  use  of  each  position 
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on  the  7-poinl  scale  (Tables  12  through  17  of  Hahn,  1975).  Only  the  first  four  entries  of  these  tables  have 
been  shown  in  Figure  2 because  inspection  of  the  data  revealed  that  these  four  *‘task  distributions”  were 
representative  of  the  entire  sets.  All  of  the  ensuing  distributions  are  suggested  in  these  comparison  graphs. 
The  graphs  show  that  less  use  of  the  rating  value  7 was  made  by  personnel  of  the  304X4  AFSC  than  by  the 
incumbents  and  raters  of  the  otiier  two  AFSCs,  and  that  most  use  of  the  7 ratings  was  made  by  AFSC 
291X0  personnel.  Performance  ratings  were  rare  below  the  midpoint  of  the  7-point  scale,  and  more  likely 
to  be  self- applied  by  incumbents  than  applied  to  them  by  other  raters.  These  graphs  answer  Question  (1) 
affirmatively:  Tliere  was  systematic  variance  in  the  scale  use. 

Figure  2 was  derived  from  Tables  1 through  6 of  the  management  report.  It  is  very  revealing  of  the 
data  and  it  answers  several  questions  through  distributions  of  mean  task  ratings.  Since  these  are 
distributions  of  means  whose  basic  n’s  were  extremely  variable,  the  tables  were  examined  to  deteniiine  the 
smallest  n.  It  proved  to  be  for  supervisors  who  rated  on  three  tasks  in  AFSC  304X4,  and  the  number  was  41 
observations.  Tire  smallest  number  of  incumbents  rating  themselves  on  a task  was  also  in  AFSC  304X4,  and 
this  number  was  77.  The  distributions  of  Figure  3 are  thus  the  means  of  task  ratings  based  on  substantial 
n’s,  some  representing  over  1,000  observations.  It  follows  that  the  stability  of  these  histograms  is 
substantial.  Tire  graphs  indicate  that  enougji  incumbents  used  the  middle  and  tlie  lower  end  of  the  scale  to 
reduce  the  mean  of  the  incumbent  distributions  markedly,  wliich  provided  greater  variance  for  incumbents. 
Ratings  in  the  304X4  AFSC  are  lower  than  those  for  the  other  two.  The  graphs  provide  tliese  answers  to 
the  initial  questions: 

1 . Tliere  was  systematic  variance  in  rating  task  performance. 

2.  Tlie  graphs  alone  could  not  describe  the  relative  performance  of  individual  tasks  — no  answer. 

3.  Some  incumbents  rated  themselves  lower  than  their  peers  and  supervisors  rated  them. 

4.  There  were  clear  differences  between  the  task  performance  rating  level  of  the  three  AFSCs,  with 
AFSC  304X4  lower  than  the  other  two. 

Question  (2)  could  not  be  answered  because  the  means  shown  in  Figure  3 could  have  been  achieved 
by  different  task  compositions,  without  stability  in  the  rating  of  any  particular  one.  Hie  questions  could  be 
answered  by  asking  whether  the  means  of  ratings  by  incumbents,  peers,  and  supervisors  for  specific  tasks 
were  converging  on  the  same  values.  Hiis  was  done  by  correlating  the  mean  value  provided  in  the 
contractor’s  Tables  1 through  6 (Hahn,  1975).  Tlie  following  results  emerged: 

AFSC  291X0;  incumbent  means  versus  peer  means,  ri  2 “ -799 

incumbent  means  versus  supervisor  means,  r,  3 = .731 
peer  means  versus  supervisor  means,  r2  3 “ .899 

AFSC  304X4;  incumbent  means  versus  peer  means,  1^2  - -590 

incumbent  means  versus  supervisor  means,  ri  3 = .516 
peer  means  versus  supervisor  means,  12  3 = .593 

A FSC  43 1 X 1 C ; incumbe nt  means  versus  peer  means , rj  2 “ -914 

incumbent  means  versus  supervisor  means,  T]  3 = .872 
peer  means  versus  supervisor  means,  r2  3 = .881 

The  question  of  convergence  is  clearly  answered:  Tliere  were  systematic  differences  in  task  difficulty  for 
each  AFSC  that  were  recognized  by  all  raters.  Additional  computations  provided  the  evidence  that  ratings 
by  peers  and  supervisors  were  converging  on  higher  values  than  were  tJiose  of  incumbents.  The  grand  means 
were  as  follows: 

AFSC  291X0;  incumbents,  5.93;  peers,  6.21 ; supervisors,  6.20; 

AFSC  304X4;  incumbents,  5,47;  peers,  5.80;  supervisors,  5.77; 

AFSC  43 IX  1C;  incumbents,  5.87;  peers,  6.06; supervisors,  6.14. 

The  contractor  has  provided  useful  interpretations  of  the  task  performance  ratings,  which  are  quoted 
from  the  management  report  (Hahn,  1975): 
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While  there  was  a lendency  tor  tneumbent  ralinys  ol  task  level  pcrroritianec  lo  pile  up  at  the 
high  end  of  the  scale  for  all  three  career  ladders,  there  w'crc  difterences  between  ladders  and  there  were 
same  itUra-task  differences  within  at  least  two  of  the  ladders.  The  reasons  for  such  dilTerenccs  are  not 
immediately  apparent.  The  tasks  within  Al  SC  291X0  tended  to  be  less  technical  and,  at  leasi  in  the 
judgment  of  the  contractor  staff,  somewhat  less  dilileulS  than  many  in  the  other  two  Al  St  s.  The 
extrente  loading  on  the  top  rating  point  tor  this  AbSC  could  relleet  a lack  ol  actual  pcrlorinanee 
variance  in  the  field.  Many  of  the  tasks  for  Al  SC’  304X4  were  of  a more  technical  nature  and  tended 
to  be  more  oriented  toward  specific  hardware  classes.  This  could  account  for  the  greater  variance  in 
obtained  ratings  for  this  Ah  SC.  bor  many  ot  the  tasks  somewhat  explicit  standards  tor  the  accuracy  ol 
task  completion  were  available  in  terms  ot  hardware  tolerances.  This  may  make  it  easier  for 
ineumbems  to  assess  task  performance  more  realistically.  It  should  also  be  noted  that  the  mean  task 
ratings  for  Ab’SC  304X4  were  lawyer  than  for  the  other  two  career  fields.  I ven  though  this  AbSC  has 
only  school  trained  input  as  opposed  to  a mixed  input  tor  AbSC.  291X0.  there  appears  to  be  more 
performance  variance  in  the  tield  and  at  least  incumbent  ratings  o!  sell -performance  rellccl  this. 

AbSC  431X1C  showed  some  of  the  extreme  high  end  loading  on  many  of  the  tasks,  but  there 
were  exceptions  on  abtsui  one  ijuartcr  ot  the  tasks  in  the  career  tieUb  This  appears  to  illustrate  that  at 
least  incumbents  do  recognize  ditlerences  in  pertormanee  ot  the  various  tasks  which  make  up  their 
jobs  and  arc  not  reluctant  to  rep^.>rt  such  ditlerences. 

The  rating  distributions  for  supervisor  and  peer  ratings  of  task  pertormanee  lend  to  relleet  the 
same  trends  found  in  the  ineumhent  ratings  for  the  three  career  fields.  . , 

The  contractor  lias  provided  the  standard  deviations  of  the  perfomiance  ratings  by  tasks  whose  means 
arc  shown  in  Figure  3.  In  general,  the  standard  deviation  of  a set  of  perfomiance  ratings  should  be  a 
measure  of  the  agreement  of  judges,  inversely,  on  the  performance  ol  a task.  The  measure  is  not  interpreted 
here  as  a measure  of  agreement  because  ot  7-point  rating  scale,  or  any  rating  scale,  yields  an  ambiguous 
standard  deviation  when  there  is  a tendency  for  raters  to  pile  responses  at  one  end  of  the  scale. 


Skill  and  Ability  Versus  Motivation  Ratings 

The  contractor  luis  provided  frequencies  converted  to  percentages  for  use  ot  tiie  five  positions  on  the 
skill  and  ability  versus  motivation  scale.  These  show  how  Irequenlly  each  position  was  used  relative  to  the 
total  ratings  made  on  a given  task  in  Tables  18  througli  20  in  the  management  report  (tiahn,  1975). 
Positions  1 and  2 were  more  frequently  used  on  this  5 -point  scale  than  were  the  lower  positions  on  the 
7 -point  scale,  which  indicates  that  at  least  a number  of  the  raters  were  following  the  intent  of  tlie 
instructions.  Ro uglily,  the  instructions  indicated  that  level  I and  2 were  not  to  be  regarded  as  qualitatively 
the  same  as  levels  3,  4,  and  5.  Hence,  the  skill  and  ability  versus  motivation  scale  was  not  strictly  a scale, 
although  it  was  formatted  as  one.  In  the  case  of  raters  who  could  not  distinguish  in  their  own  minds 
whether  a ratee’s  less-th an -optimal  task  iierformance  was  due  to  inexperience  or  motivation,  the  scale 
format  may  liave  offered  a chance  to  repeal  the  task  performance  rating  without  contributing  new 
information.  Tliis  is  an  unsettled  question  which  is  suggested  by  the  contractor’s  interpretations,  most  of 
which  are  quoted  below.  It  is  probable  that  the  use  of  the  skill  and  ability  versus  motivation  scale  was  not 
consistent  from  ralcr  to  rater.  AFHRL  has  treated  the  data  as  though  a continuous  scale  were  being  applied 
by  correlating  the  ratings  with  those  on  the  7 -point  task  perloriiiance  scale  tor  the  same  tasks.  Coefficients 
around  .80  appeared,  which  suggested  tliat  a number  of  the  raters  may  have  been  merely  repeating  their 
ratings  on  a coarser  scale.  (However,  there  are  indications  arising  from  the  analyses  that  poor  motivation 
was  perceived  by  raters,  and  that  their  ratings  rcficctcd  it.  Also,  motivation  was  the  most  common  criticism 
made  in  those  cases  wliere  raters  completed  the  comments  page  ol  the  rating  booklet,)  Tlie  management 
report  (Hahn,  1975)  says: 

. . . ll  WLis  biised  upon  Ihu  concepi  otdeficil  between  poienlial  eapaeiiy  and  actual  performance. 

The  total  scale  was  not  truly  linear.  Point  5 indicated  essentially  no  deficit.  Paint  4 indicated  a small 
deficit  due  almosi  entirely  lo  training  needs.  Points  3 and  2 indicated  a small  delicil  attributable  to  a 
mixture  of  skill  and  motivational  factors.  Point  I indicated  a greater  deficit  due  almost  entirely  lo 
motivational  factors.  Por  some  correlational  analyses  rctxirlcd  later,  a somewhat  risky  assumpiion  of 
linearity  was  made  based  on  a tenuous  continuum  of  greatest  deficit  lo  no  deficit  with  the  somcwhal 
tacit  assumption  that  deficits  attributable  to  motivational  factors  were  somewhat  less  desirable  than 
those  attributable  to  training  needs.  . . 

Review  of  these  tables  indicates  that  far  Al  SC  29  1 XO  and  A I- SC  43  I XIC  rating  point  5 was  the 
modal  point  for  all  task  dimensions  for  supervisors  and  peers.  . . The  results  for  AbSC  304X4  were  in 
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llic  same  dlrculion  but  not  quite  a?i  extreme,  , , , ThejtC  data  in  conjunction  witk  the  generally  lower 
nieim  task  pcrronnance  ratings  reported  previously  for  this  career  ticid  tend  to  indicate  that  there  may 
be  a somewhat  greater  need  lor  remedial  action  in  this  Held  than  in  the  other  two.  Since  the  aptitude 
input  tor  this  field  is  extnal  to  or  higher  than  that  of  the  other  two  career  fields,  the  explanation  for 
these  sliglit  differences  must  lie  elsewhere.  As  indicated  previously,  the  tasks  for  this  career  I’veld  tend 
to  be  highly  technical  and  related  to  specific  hardware  items  for  many  of  which  there  arc  relatively 
ckise  specified  operational  tolerances.  Although  the  data  do  not  supply  a detinitivc  ans-wer,  it  is  the 
feeling  of  the  contractor  staff  that  such  characteristics  probably  account  for  the  obtained  differenc'cs. 

The  rating  point  used  second  mosl  frequently  in  all  three  career  fields  was  4 which  indicated 
some  deficit  due  largely  to  training  needs.  . . . 

Only  a relatively  small  percentage  of  jxr  forma  nee  deficits  were  attributable  largely  to 
moiivalionai  factors.  1 or  Al  SC  291X0  the  percentages  for  the  combined  I and  2 categories  rangeti 
trom  0 to  7 lor  both  supervisors  and  peers,  i or  Al  SC  304X4  these  percentages  ranged  from  1 to  14 
for  supervisors  and  from  2 to  20  for  peers,  I or  Al  SC  431XIC  the  percentages  ranged  from  2 to  1 3 for 
supervisors  and  from  1 to  13  for  peers. 

Later,  after  making  correlational  analyses,  the  contractor  says; 

. . I rom  the  skill  and  ability  versus  motivation  ratings,  it  is  clear  that  the  majority  of 
supcrvissiTs  and  txers  feel  that  a large  penx'ntage  of  incumbents  in  the  three  fields  are  usually  working 
close  to  their  potential  capacity.  When  deficits  were  reported,  these  were  more  often  attributed  solely 
to  skill  deficits  or  a combination  of  skill  and  motivation  ileficits  than  they  were  solely  to  motivational 
deficits  or  motivationally  dominated  mixtures  with  skill  dcUcits.  Slightly  greater  dellcits  were  noted  in 
Al  SC  304X4  than  for  the  other  two  ciu^cer  fields. 

Correlations  of  Task  Perfomiance  Ratings 
Made  on  the  Same  Incumbents 

The  premier  question  of  this  research  is  whether  or  not  judges  of  incumbent's  job  performance  will 
agree  in  their  appraisals  of  separate  task  performances.  If  one  assumes  that  the  answer  is  YES,  he  will  then 
need  to  consider  several  specific  issues.  These  are  outlined  immediately  below  to  identify  the  content  of 
this  section,  but  it  is  desirable  to  digress  a little  before  presenting  the  answers,  Tlie  issues  are: 

1,  Wlial  is  the  distribution  of  the  measures  of  agreement? 

2,  Axe  there  systematic  differences  among  the  three  AFSCs  with  respect  to  agreement  on  task 
performance  ratings? 

3,  Is  one  kind  of  rater  measurably  superior  to  another,  such  as  a peer  being  a better  judge  than  a 
supervisor? 

4,  Do  incumbents  contribute  usable  data  on  self-ratings  of  task  performance? 

5,  Do  the  data  yield  any  clues  that  will  differentiate  tasks  with  respect  to  the  ratability  of  their 
performance? 

Some  of  these  points  were  covered  in  the  course  of  preliminary  analyses  and  the  decision  was  made 
not  to  pursue  them  further.  The  direction  which  the  analyses  took  was  partly  determined  by  computations 
undertaken  by  AFHRL  on  the  same  data  bank  as  used  by  AIR,  and  concomitantly,  Tliese  produced  the 
discovery  tJiat  certain  approaches  would  not  be  likely  to  be  productive.  They  are  mentioned  here  so  that 
other  investigators  can  avoid  efforts  which  might  prove  to  be  wasted.  It  should  be  kept  in  mind  that  when 
these  analyses  were  begun,  there  existed  no  guidelines  as  to  wliat  one  should  expect  to  find  in  operational 
ratings  of  task  performance. 

It  is  normally  assumed  that  psychological  measures  converge  on  stable  values  when  more  observations 
are  added  to  their  computed  mean.  With  this  concept,  it  would  be  assumed  that  if  one  correlates  the  task 
performance  ratings  made  on  an  incumbent  by  a peer  and  a supervisor,  then  goes  to  the  data  bank  to  find  a 
third  rater,  he  could  expect  the  third  rater  to  stabilize  the  data.  Me  would  expect  that  if  he  computes  the 
mean  of  the  peer  and  supervisor  ratings,  then  correlates  the  new  rater  with  (a)  the  peer,  (b)  the  supervisor, 
and  (c)  the  mean  of  the  peer  and  siJ|x;rvisor's  ratings,  tlic  third  correlation  w ould  be  the  higliest,  Tliis  was 
done  and  it  was  continued  out  to  four  raters  where  available.  While  all  the  correlation  coefficients  were 
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rather  unstable,  correlations  with  means  of  two  raters  were  slightly  luglier  than  correlations  between  single 
raters*  Now  a new  mean  of  three  ratings  was  available^  and  all  possible  combinations  were  computed.  The 
net  result  was  that  there  was  no  practical  increase  in  the  set  of  correlation  coefficients  beyond  computing 
the  mean  of  two  raters,  almost  any  two  who  rated  a single  task  performance  of  an  incumbent.  This  could 
have  been  a function  of  special  operational  situations,  since  only  certain  types  of  tasks  could  be  perfonned 
by  enough  people  to  provide  several  observers  of  the  same  incumbent.  Bearing  in  mind  that  additional 
raters  must  be  equally  able  to  observe  the  incumbent's  work,  the  failure  to  increase  reliability  may  have 
been  a function  of  adding  random  variance  to  an  unstable  set  of  data,  where  to  achieve  a stable  value  would 
require  more  raters  than  could  exist.  From  the  experimenter’s  standpoint:  in  this  case  two  raters  were 
sufficient  and  procuring  additional  raters  miglit  not  be  cost  effective. 

As  one  adds  raters  to  stabilize  the  task  performance  rating  mean  of  a specific  incumbent  on  a specific 
task,  he  is  not  doing  the  same  thing  as  was  done  to  produce  the  data  of  Figure  3.  Tlie  means  of  task  ratings 
for  all  raters  performing  a given  task  appear  to  behave  as  do  most  accepted  statistics,  converging  on  stable 
values.  In  the  previous  correlations  between  task  means  obtained  from  incumbents,  peers,  and  supervisors 
for  AFSC  304X4  it  was  found  that  the  coefficients  were  lower  than  tliosc  for  the  other  two  AFSCs,  It 
seems  probable  that  the  reason  was  the  fact  that  raters  in  AFSC  304X4  used  the  7 value  of  the  scale  less 
than  the  raters  in  the  other  two  AFSCs,  which  left  instability  in  the  upper  end  of  tlte  scale  as  well  as  the 
lower.  Put  another  way,  there  was  room  for  individuals  to  vary  at  the  top  of  the  AFSC  304X4  task 
performance.  We  shall  see  that  this  may  have  contributed  to  making  the  performance  of  these  personnel 
much  more  measurable  than  was  that  of  AFSCs  291X0  and  43 1 XI C, 

It  turned  out  from  these  early  analyses  that  incumbent  self-ratings  correlated  less  well  with  ratings 
assigned  to  them  by  either  peers  or  supervisors  than  the  peer  and  supervisor’s  ratings  correlated  with  each 
other.  It  was  evident  that  the  most  fruitful  data  would  come  from  observers  who  were  objective  rather  than 
subjective.  One  should  note  that  the  matter  of  objectivity  was  primarily  the  role  that  an  individual  was 
playing,  since  an  incumbent  became  a peer  or  a supervisor  when  he  was  rating  someone  else. 

The  contractor  pursued  the  problem  of  peer  and  supervisor  agreement  in  detail,  reporting  on  task 
performance  ratings  and  the  skill  and  ability  versus  motivation  ratings  in  Tables  22  througli  24  of  the 
management  report  (Halm,  1975).  He  gives  the  number  of  incumbents  who  rated  on  each  task  along  with 
the  peer  and  supervisor  correlations  for  both  the  task  performance  and  the  skill  and  ability  versus 
motivation  scales.  Accompanying  these  is  the  probability  estimate  that  an  obtained  coelficient  could  have 
occurred  by  chance  based  upon  die  likelihood  that  in  1 ,000  trials  tlie  coefficient  could  have  been  a 
deviation  from  a true  r of  zero.  Tliis  is  a strict  application  of  the  probability  assumptions  because  it  treats 
each  coefficient  as  though  it  were  independently  obtained.  Tlie  probability  tiiat  the  whole  distribution 
would  have  varied  from  zero  in  the  positive  direction  is  not  considered. 

Tlie  skill  and  ability  versus  motivation  ratings  will  be  disposed  of  first  by  dismissing  them.  Tlicsc 
ratings  probably  had  validity  for  a number  of  raters,  but  the  distribution  of  correlations  suggests  that  they 
were  a less  reliable  reflection  of  the  task  performance  ratings  which  immediately  preceded  them.  For  the 
most  part,  the  coefficients  in  the  skill  and  ability  versus  motivation  column  are  smaller  than  their 
corresponding  task  performance  rating  conelations,  and  they  follow  the  same  pattern.  It  should  not  be 
inferred  from  this  that  the  concept  of  rating  skill  and  ability  versus  motivation  was  invalid;  the  difficulty 
lay  in  a combination  of  format  and  proximity  to  a rating  just  made.  Tlie  concept  should  still  be  tested, 

Tlie  correlations  provided  by  die  contractor  answer  the  basic  question  of  how  well  raters  agreed  on 
evaluating  task  performance.  Tiicse  have  been  reduced  to  graphic  form  in  Figure  4.  Tlie  illustration  is  three 
sets  of  bar  graphs,  the  upper  tier  representing  the  cocfficicnls  obtained  when  less  than  100  incumbents  were 
rated  on  the  pcrfomiance  of  a task,  the  middle  tier  representing  coefficients  for  100  to  199  incumbents, 
and  tlie  lowest  tier  representing  correlation  coefficients  for  200  or  more  ratees.  Tlie  baseline  places 
correlation  coefficients  in  intervals  of  .05  and  the  frequencies  are  shown  in  units  of  1 . Bar  graphs  appear  for 
each  AFSC  within  an  interval  in  die  following  order:  291X0,  304X4,  and  431X1C.  A dashed  line  has  been 
drawn  through  one  of  the  intervals  in  each  tier,  specifically,  r=  .325,  r=  .225,  and  r=  .175.  Tlicse  are 
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approximate  values  for  correlations  different  from  zero  at  the  .01  confidence  level,  based  respectively  upon 
50  observations,  150  observations,  and  250  observations  per  correlation.  The  bar  graphs  represent  the  peer 
versus  supervisor  correlations  in  the  task  performance  rating  columns  of  Tables  22  through  24  of  die 
managenient  report  (Halm,  1975).  Although  the  dashed  lines  are  a rough  approximation,  by  actual  count  of 
the  contractor’s  r’s  with  computed  significance  at  the  .01  level  or  better  there  is  very  little  difference 
between  the  number  to  the  right  of  the  dashed  line  and  the  number  of  significant  r’s  wtuch  the  tables 
provide.  Tlie  number  of  coefficients  in  Figure  4 depends  upon  the  number  of  tasks  rated,  which  was  51  for 
AFSC  291X0, 92  for  AFSC  304X4,  and  55  for  AFSC  431X1C, 

Returning  to  the  questions  at  issue  - (1)  What  is  the  range  and  frequency  (distribution)  of  the 
measures  of  agreement?  -it  is  seen  that  the  size  of  the  correlations  is  a function  of  the  AFSC.  As  a set  of 
statistics,  the  correlations  for  AFSC  304X4  could  not  have  arisen  from  the  same  population  as  those  of  the 
other  two  AFSCs  within  any  reasonable  probability.  Only  5 of  the  92  AFSC  304X4  coefficients  are 
nonsignificant,  16  of  the  55  coefficients  for  AFSC  43!XiC  are  nonsignificant,  and  33  of  the  51  coefficients 
for  AFSC  291X0  are  nonsignificant.  These  data  answer  question  (2).  Are  there  systematic  differences 
between  the  AFSCs  with  respect  to  agreement  on  task  performance  ratings?  Collapsing  the  three  tiers  into 
one  distribution  for  each  AFSC,  one  finds  that  the  median  r for  AFSC  304X4  lies  in  the  interval  .30— .35 
and  that  the  medians  for  both  AFSC  291X0  and  43 1 XI C lie  in  the  interval  .15-. 20.  Without  qualification, 
AFSC  304X4  is  superior  to  the  other  two  specialties  in  task  perfonnance  rating  agreement,  and  AFSC 
431X1C  appears  to  have  had  more  ratable  tasks  than  AFSC  291X0. 

Tills  section  of  the  analyses  was  begun  with  the  assumption  that  raters  could  agree  upon  task 
performance  evaluations.  Had  they  not  been  able  to  agree  to  some  extent,  the  bars  of  Figure  4 would  have 
clustered  about  the  value  zero. 

Tlie  last  three  questions  raised  in  the  analyses  of  rater  agreement  will  be  given  tentative  answers 
without  data  to  support  them  at  this  point.  Question  (3)  — is  one  kind  of  rater  measurably  superior  to 
another?  — was  answered  partially  by  the  preliminary  analyses  performed  by  AFHRL  when  it  was  found 
that  peer  ratings  correlated  as  well  with  pooled  ratings  as  did  supervisor’s  ratings  but  that  incumbents 
self-ratings  yielded  lower  correlations.  Additional  data  were  amassed  when  computing  other  relationships, 
which  suggested  that  in  most  respects  supervisors  were  more  reliable  sources  of  judgments  than  were  peers. 
By  that  time  analyses  of  incumbent  data  had  been  dropped.  Incumbents  could  not  be  correlated  with 
incumbents  as  peers  could  be  with  peers  and  supervisors  with  supervisors.  If  the  incumbent  was  providing 
valid  unique  data,  that  fact  would  have  to  be  reached  in  a different  manner.  Tlius,  question  (4)-  Do 
incumbents  contribute  usable  data  on  self-ratings  of  task  performance?  - has  not  actually  been  answered, 
and  it  has  not  been  eliminated  from  future  consideration.  Finally,  question  (5)  - Do  the  data  yield  any 
clues  that  will  differentiate  tasks  with  respect  to  the  ratability  of  their  performance?  - was  noted  by  the 
contractor.  Tliis  question  is  the  essence  of  the  research  problem  and  it  will  be  dealt  with  later. 

Prediction  of  Task  Performance  Ratings  from 
Data  of  Record  and  Cognitive  Test  Scores 

Tlie  justification  for  the  entire  enterprise  is  to  show  that  rating  task  perfomiance  contributes  to 
understanding  whole  job  performance.  This  requires  that  task  performance  ratings  be  predictable  or 
accountable  from  something.  That  is,  do  tasks  in  one  specialty  reflect  aptitude,  while  tasks  in  another 
specialty  refiect  other  factors,  such  as  length  of  service,  social  skills,  attitudes,  grade,  or  some  aptitude  as 
yet  unmeasured?  Or  do  both  sets  of  task  ratings  show  no  predictability  from  anything  that  is  consistently 
measurable? 

Obviously,  one  must  dispense  with  this  last  possibility  before  he  needs  to  trouble  himself  about  the 
others.  Tlie  contractor  has  provided  relevant  analyses  in  a predictor-by^predictor  discussion  in  the 
management  report  (Hahn,  1975).  Tliese  analyses  have  been  summarized  in  Table  2:  it  may  be  well  to  point 
out  how  the  data  have  been  assembled  to  produce  its  cell  entries,  for  it  is  the  product  of  a number  of  steps. 
A great  deal  of  data  were  collected  in  the  surveys,  and  these  data  were  augmented  by  giving  the  contractor 
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Table  2,  Significant  Correlations  with  Task  Performance  Rating? 


Predictor 

AFSC  231X0 
51  tasks) 

AFSC  304^X4 
(Base  92  tasks) 

AFSC  431  XtC 
(Base  5S  tasks) 

Peer 

Supvr.  Comp. 

Peer 

Supvr,  Comp. 

Peer 

Supvr.  Comp. 

Grade 

15 

13 

18 

61 

43 

52 

22 

35 

20 

Monllis  on  base 

13 

25 

27 

2 

14 

10 

2 

13 

8 

Months  in  AFSC 

9 

9 

8 

58 

3 

29 

4 

19 

6 

TAFMS^ 

6 

9 

8 

57 

9 

47 

6 

11 

5 

Decoding  score 

3 

0 

0 

0 

6 

3 

3 

3 

M Cl  n 0 ry  fo  r Ian  dm  ark  s 

7 

5 

4 

6 

24 

3 

4 

I 

Complex  scale  reading 

7 

! 

9 

3 

0 

3 

1 

11 

Pursuit 

3 

0 

3 

1 

1 

1 

1 

9 

11 

Figure  analogies 

9 

5 

8 

3 

18 

2 

1 

9 

1 

Mands 

3 

3 

0 

1 

8 

4 

3 

3 

1 

Cubes 

1 

1 

0 

5 

5 

10 

2 

0 

6 

Mechanical  principles 

0 

0 

0 

15 

42 

32 

5 

2 

5 

Following  directions 

2 

0 

1 

4 

13 

3 

1 

1 

2 

Practical  estimations 

S 

9 

3 

0 

16 

9 

7 

3 

Spatial  reasoning 

1 

1 

1 

7 

9 

3 

3 

Marital  status 

6 

9 

10 

6 

6 

4 

0 

1 

I 

Size  of  city  of  origin 

i 

3 

4 

2 

2 

2 

1 

15 

4 

Mechanical  Al 

3 

11 

1 

10 

2 

3 

15 

5 

9 

Administrative  AI 

1 

24 

3 

3 

T 

1 

0 

2 

2 

General  AI 

0 

5 

1 

3 

I 

2 

1 

2 

Flectrcnic  Al 

1 

8 

0 

8 

12 

15 

2 

0 

0 

Sex 

2 

6 

6 

- 

- 

- 

Year  of  enlistment 

5 

12 

12 

50 

15 

24 

8 

45 

10 

Education  level  of  enlistment 

2 

4 

0 

71 

23 

57 

20 

35 

14 

APQT  score 

2 

7 

2 

4 

1 

1 

3 

1 

1 

Mean 

4.4 

6.S 

5.2 

15.2 

10.8 

12,9 

4,5 

8.7 

5.2 

Mean  in 

8.6 

13.3 

10.2 

16.5 

1 L7 

14.0 

8.2 

15.8 

9.5 

^TAFMS  aciivt;  ftrdvral  inillury  it^rvicL^ 


mformation  obtained  from  /or  Force  record  tapes  maintained  by  AFFIRL  Incumbent  names  and  Social 
Security  numbers  were  matched  with  the  tape  records.  Rather  than  enumerate  all  the  items  collected  from 
official  records,  the  ones  that  were  differently  obtained  are  cited.  The  1 1 cognitive  tests  were  part  of  the 
surveys.  Tiie  months  on  base  and  months  in  AFSC  were  taken  from  the  job  inventory  information  supplied 
by  the  incumbent.  Months  on  base  cannot  be  extracted  accurately  from  the  official  records  Files  because 
leave  time  and  other  inconsistencies  make  real  time  on  base  and  official  lime  different.  Cumulative  Items, 
such  as  time  in  AFSC,  are  better  given  by  the  incumbent,  for  similar  reasons.  Using  predictor  variables 
which  employed  Job  inventory  data  automatically  excluded  incumbents  sampled  in  the  supplemental 
survey,  since  that  survey  did  not  administer  Job  inventories.  Tlie  use  of  cognitive  test  score  predictors  meant 
that  any  personnel  who  failed  to  take  the  battery  were  excluded.  Lack  of  experimental  test  score  data  was 
the  largest  single  source  of  data  attrition  in  the  entire  study.  To  round  out  the  list  of  selective  effects  of 
predictor  variables,  the  size  of  city  of  origin  was  an  item  in  the  Biograpirical  Inventory,  one  of  the 
noncognitive  instruments  of  the  experimental  battery.  Tlie  remaining  predictors  listed  in  Table  2 came 
either  from  official  record  tapes  or  were  corroborated  by  the  records.  These  were  quite  complete  and  did 
not  represent  a factor  wliich  could  select  incumbent  data. 

Thus,  from  predictor  to  predictor  there  were  conditions  which  affected  the  sampling  of  ratees  in  each 
correlation  coefficient,  Tliis  does  not  cover  the  sampling  of  raters,  where  such  factors  as  shift  work  and  the 
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kind  of  task  performed  operated  to  select  raters.  In  computing  a correlation  coefficient  the  contractor 
appears  to  have  tried  to  maxirmze  the  number  of  observations  in  the  criterion  vector.  This  was  legitimate 
and  desirable,  but  it  must  be  appreciated  in  order  to  interpret  the  data  of  Table  2.  The  number  of  peers  or 
supervisors  rating  an  incumbent  on  a specific  task  was  so  variable  that  it  is  inconceivable  that  any  two 
coefficients  would  have  depended  upon  precisely  the  same  ratees  and  raters.  Thus,  the  cell  entries  of  Table 
2 must  represent  counts  of  correlations  with  extremely  variable  numbers  of  observations.  Tliese  coefficients 
were  small,  though  usually  positive,  probably  ranging  from  —.17  to  .27,  but  having  fairly  substantial  n’s. 
They  could  consequently  be  identified  as  significantly  different  from  a true  coefficient  of  zero,  and  this  was 
reported  at  the  5 percent  level  of  confidence.  The  entries  of  Table  2 are  a simple  count  of  occurrence  for 
each  ceil.  These  depend  upon  the  number  of  tasks  rated  in  each  specialty,  51  tor  AFSC  291X0,  92  for 
AFSC  304X4,  and  55  for  AFSC  431X1C.  For  all  practical  purposes  the  count  of  cells  for  AFSC  291X0  can 
be  compared  directly  to  the  count  for  AFSC  431X1C;  and  to  provide  a rougli  approximation,  the 
equivalent  count  for  AFSC  304X4  can  be  obtained  by  dividing  the  cell  entry  by  2. 

As  the  contractor  has  pointed  out  in  his  summary  of  these  analyses,  I in  20  of  the  coefficients  could 
appear  as  significant  from  chance  occurrences.  The  table  reveals  that  some  predictors  yielded  frequencies 
far  above  chance  for  all  tliree  specialties,  grade  and  year  of  enlistment  being  notable  producers  of 
nonchance  coefficients.  Both  of  these  predictors  reflect  time  in  service,  and  year  of  enlistment  also  reflects 
the  chronological  age  of  the  incumbent.  As  one  looks  at  the  other  predictor  variables,  it  becomes  difficult 
to  determine  which  are  predictive,  or,  if  any  really  were.  However,  the  table  provides  an  answer  to  the  most 
serious  doubt  raised;  some  predictors  did  make  statistically  significant  predictions  of  task  perfonnance 
ratings.  Furthermore,  enougli  nonchance  predictions  occurred  to  indicate  that  several  variables  were 
correlated  with  task  performance  beyond  chance.  The  mean  number  of  nonchance  correlations  was 
computed  for  each  column,  and  in  the  bottom  row  it  is  shown  as  a percent.  The  values  obtained  exceed  the 
5 percent  limit,  and  even  when  grade  is  removed  and  the  mean  is  recomputed,  the  percent  values  exceed 
chance  expectation. 

The  prediction  of  the  composite  ratings  shown  in  the  third  column  lies  between  that  of  the  low  and 
high  group  for  each  AFSC.  Wliile  this  would  seem  to  be  a simple  arithmetic  necessity,  in  fact  it  is  not.  Tlie 
composites  are  made  up  of  two  raters,  wMch  eliminates  unpaired  cases  from  the  two  columns  preceding  and 
greatly  reduces  the  number  of  observations.  This,  in  turn,  lowers  the  probability  of  a significant  correlation 
coefficient  appearing. 

In  the  29 1X0  and  43 1X1 C AFSCs  the  larger  number  of  nonchance  correlations  arose  from  supervisor 
ratingSi  and  in  the  304X4  AFSC  the  peer  ratings  produced  more  nonchance  fs.  A similar  observation  was 
found  in  subsequent  analyses,  suggesting  that  some  factor  (such  as  opportunity  to  observe  the  ratee)  may 
have  operated  to  differentiate  the  AFSCs. 

At  best,  the  data  of  Table  2 indicate  trends.  Tlie  interdependency  of  such  variables  as  length  of 
service  and  grade  require  that  regression  approaches  be  used  to  determine  their  unique  value  as  performance 
predictors.  The  analyses  were  perfonned  and  are  presented  later. 

Correlations  Among  the  Six  Overall 
Performance  Rating  Dtinensioris 

Six  overall  performance  ratings  were  made  by  peers  and  supervisors  upon  completion  of  tlie  task 
dimension  ratings.  Since  the  number  of  task  dimensions  was  different  for  each  AFSC,  and  represented  an 
appreciable  reading  time,  the  number  of  ratings  made  before  the  six  final  ratings  differed  considerably  for 
the  three  specialties.  Task  ratings  were  successive  acts,  involving  marks  on  pages  of  the  survey  booklet  and 
sufficient  reading  of  each  for  the  rater  to  decide  whether  or  not  he  would  rate  performance  on  a particular 
task.  The  92  (actually  95)  task  dimensions  of  AFSC  304X4  represented  nearly  twice  the  reading  time  of  the 
5 1 task  dimensions  of  AFSC  291X0.  On  the  other  hand,  the  six  overall  performance  rating  dimensions  aU 
appeared  on  die  same  page  of  the  booklet,  and  their  reading  time  was  relatively  brief.  Consequently,  one 
would  expect  that,  if  the  raters  were  performing  in  a perfunctory  way,  the  correlations  among  the  six  scales 
would  be  very  high.  They  are  shown  in  Table  3 to  have  been  iiigh,  but  not  as  inflated  as  one  might  expect 
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Table  3,  Correlations  Among  Six  Overall  Terfomiance  Rating  Dimensions 


Ratings  By  Peers 

Ratings  By  Supervisors 

AFSC 

Dimension  OP  Quan  Qual  FS 

St 

SK 

GP 

QuaL 

F5  5E 

SK 

554— 632  incumbents  rated 

796—853  incumbents  rated 

291X0 

General  performance  (GP) 

.82  .85  .78 

.81 

,74 

.87 

.87 

.79  .81 

.78 

Quantity  of  w'ork  (Quan) 

.80  .80 

.81 

.71 

.86 

.80  .82 

.77 

Quality  of  work  (Qual) 

.76 

.78 

.72 

.74  .77 

.74 

Exceeds  his  share  (ES) 

.88 

.76 

.92 

.78 

Self-initiating  (SI) 

Shares  knowledge  (SK)- 

.79 

.82 

371— 419  incumbents  rated 

420—447  incumbents  rated 

304X4 

General  performance  (GP) 

.82  .85  .74 

.78 

,72 

.89 

.86 

.81  *83 

.80 

Quantity  of  work  (Quan) 

.72  .78 

.75 

.64 

.83 

.84  .84 

.76 

Quality  of  work  (Qual) 

.67 

.71 

.68 

,76  .80 

.76 

Exceeds  his  share  (ES) 

*85 

.66 

.89 

.77 

Sell -initialing  (SI) 

Shares  knowledge  (SK) 

,68 

.80 

539—619  incumbents  rated 

747—796  incumbents  rated 

43IX1C 

General  performance  (GP) 

.87  .85  .79 

.80 

.76 

.86 

.87 

.80  .84 

.80 

Quantity  of  work  (Quan) 

.79  .82 

.78 

.75 

.84 

.84  .83 

.80 

Quality  of  work  (Qual) 

*75 

.77 

.73 

.79  ,81 

.78 

Exceeds  his  share  (ES) 

,87 

.73 

.91 

,83 

Self-initiating  ( SI) 

Shares  knowledge  (SK) 

.77 

.85 

Note:  Number  o(  r.rtiiigii  tor  LorreUdon  matrices 

a)  291X0  peer  - 5?4-632  incumbents 

b)  291X0  iiupv  “ 796-853  incumbents 

c)  3IMX4  peer  = 371-419  incumbents 

d)  304X4  supv  = 420-447  irKumbenti 

e)  431  XlC  peer  - 539  619 

h 

f)  431X1C  supv  ^ 747-796  incumbonti 

from  repetition  of  a single  overall  in^prcssion*  Table  3 is  not  strictly  an  inlercorrelation  matrix.  It  is  a 
combination  of  the  contractor’s  Tables  28  and  29  in  the  management  report  (Hahn,  1975).  Tire  ii’s  of  the 
cells  of  Table  3 vary  slightly,  and  different  incumbents  are  represented  among  the  ratees  by  peers  and 
supervisors.  However,  the  basic  sample  underlying  these  correlation  coefricients  is  a large  unit,  ajid  the 
values  shown  are  quite  similar  to  those  found  by  AFHRL  for  samples  reduced  to  the  same  incumbents*  The 
coefficients  are  not  so  large  as  to  preclude  interpretablo  differences  among  tiie  dimensions  of  overall 
perfonnaiice. 

During  preliminary  aiudyses  of  these  data  AFHRL  segregated  samples  containing  all  six  ratings  by 
peers  and  supervisors  on  the  same  incumbents*  Tlie  samples  ranged  from  300  to  500  ratees  per  AFSC.  Using 
regression  techniques  in  which  it  was  assumed  that  the  general  overall  rating  was  the  criterion,  the  other  five 
ratings  v/erc  made  to  account  for  the  general  performance  rating*  Tlie  intercorrelation  matrix  produced 
correlations  in  the  low  *80’s  when  the  rater’s  own  scales  were  correlated,  which  is  very  similar  to  what 
appears  in  Table  3.  When  peers  were  correlated  with  supervisors,  the  f"s  were  in  the  mid-  *30’s*  Employing 
the  peers’  general  overall  rating  as  criterion,  tlie  supervisor’s  five  subscaJe  ratings  were  used  as  predictors, 
and  then  the  supervisofs  general  rating  was  used  as  criterion  and  the  peers’  subscaie  ratings  were  the 
predictors*  Tire  results  were  suggestive  of  untapped  relationships  among  die  six  overall  scales*  The 
regressions  indicated  that  the  three  specialties  might  be  dissimilar  with  respect  to  the  subscales  and  their 
relative  weights  in  accounting  for  overall  or  general  performance  assessments. 
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In  sum,  the  overall  ratings  were  higlily  related  to  each  other  when  taken  from  the  same  page  of  a 
rating  booklet.  Part  of  this  correlation  would  necessarily  be  attributable  to  rater  tendency  to  use  the  top  of 
the  100-poinl  overall  rating  scale,  or  not  to  use  it.  Removing  the  rater  tendency  factor  greatly  reduced  the 
correlations,  and  the  regression  problems  provided  suggestions  that  there  were  complex  relationships  among 
the  six  ratings  which  reflected  differences  among  the  specialties.  This  was  an  important  lead  in  terms  of  its 
effect  upon  the  course  of  subsequent  analyses. 

Correlations  Between  Task  Performance  Rating 
and  Overall  Performance  Dimension  Ratings 

The  contractor  has  provided  tables  of  correlations  between  task  performance  ratings  and  the  six 
overall  dimensions  of  performance  for  both  peers  and  supervisors.  He  has  included  identification  of  those 
correlations  which  fail  to  be  significant  at  the  .01  level  of  confidence.  They  are  Tables  30  through  35  in  the 
management  report  (Hahn,  1975).  Tliese  have  been  reduced  to  graphic  form,  and  they  appear  as  Figures  5 
and  6.  Figure  5 contains  the  general  overall  rating,  the  quantity  of  work  rating,  and  the  quality  of  work 
rating.  The  three  ratings  in  Figure  6 can  be  regarded  as  motivational.  Correlations  between  peer  estimates 
are  shown  as  solid  lines  and  correlations  between  supervisor  estimates  as  broken  lines.  The  graphs  indicate 
that  in  all  respects  of  agreement  between  task  ratings  and  overall  ratings  personnel  in  the  304X4  AFSC 
provided  higher  correlations  than  did  tliose  in  the  other  two  specialties,  with  modal  fs  10  points  higher 
than  those  of  the  other  two  specialties.  AFSC  291X0  yielded  the  lowest  agreement,  though  this  was  still  in 
the  modal  range  of  r = .50.  A second  feature  of  these  graphs  is  that,  with  a lone  exception,  the  correlations 
obtained  from  supervisors  were  higlier  than  tliose  obtained  from  peers.  The  exception  occurs  in  the 
quantity  of  work  performed  in  the  291X0  AFSC.  It  is  interesting  because  it  is  one  of  the  few  instances  for 
which  an  explanation  can  be  hazarded.  The  hypothesis  is  that  this  difference  is  genuine  and  reflects  a 
difference  of  opportunity  between  peers  and  supervisors  to  observe  incumbents  in  the  291X0  specialty 
activities.  Continuous  records  are  maintained  in  a communications  center  of  messages  sent  and  received,  of 
circuit  usage,  overloading,  etc.  Peers  tend  to  move  together  on  sliifts,  wliich  do  not  correspond  as  closely  to 
the  assignment  of  supervisors.  This  provides  a peer  with  a better  estimate  of  the  incumbenCs  normal 
production.  If  the  explanation  should  prove  to  be  valid,  it  would  attest  to  the  sensitivity  of  the 
methodology. 

Summary  of  Findings  for  Unrestricted  Samples 

The  contractor  made  an  effort  to  maximize  the  size  of  the  samples,  and  for  that  reason  the  products 
of  the  contractor’s  analyses  have  been  given  precedence  for  presentation  over  parallel  analyses  performed 
by  AFHRL.  At  this  point  and  hereafter,  the  analyses  reported  will  be  those  performed  by  AFHRL 
Preparation  of  siunples  for  regression  analyses  required  selection  of  data  so  that  all  required  variables  would 
present  complete  data  for  intercorrelation  purposes.  This  greatly  limited  the  cases  that  could  be  used 
because  there  were  many  concomitant  variables  for  each  incumbent.  Tlius,  the  foregoing  analyses  could 
better  reflect  the  variables  from  which  the  data  of  the  regression  problems  were  taken,  and  it  becomes  the 
responsibility  of  the  regression  analyst  to  show  that  his  samples  were  representative  of  the  full  population 
of  the  study.  Tlie  findings  up  to  this  point  show  that: 

I-  Task  perfomiance  in  each  AFSC  could  be  rated  with  greater  than  chance  agreement  between 

raters. 

2.  Tlie  specialty,  AFSC  304X4,  which  the  contractor  considered  to  have  the  best  established  criteria 
of  task  performance  in  an  objective  sense,  yielded  the  least  use  of  the  top  of  the  rating  scale  and  the  highest 
correlations  between  raters  on  all  performance  estimates.  It  also  yielded  tlie  liigliest  within-rater 
correlations. 

3.  AFSC  29 1X0,  with  tasks  which  the  contractor  considered  likely  to  be  the  easiest  to  learn,  yielded 
the  greatest  use  of  the  top  of  the  performance  rating  scale  and  the  lowest  correlations  between  raters. 
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Figwe  5.  Correlations  of  Task  and  General  Performance  Ratings,  Product. 
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Figure  6.  Correlations  of  Task  and  General  Performance  Ratings,  Motivation. 


4.  Incumbents  frequently  rated  llicnisclvcs  lower  on  task  performance  than  they  were  rated  by  their 
peers  or  supervisors.  Tlie  uncertainties  raised  by  their  data  put  their  ratings  in  doubt  as  a source  of  useful 
performance  estimates. 

5.  Tlicre  were  systematic  differences  within  specialties  among  task  pcrfomitmcc  ratings,  as  well  as 
clear  differences  in  mean  task  per fo nuance  ratings  among  the  three  specialties. 

6*  The  AFSC  whose  incumbents  had  the  higliest  aptitude.  304X4,  yielded  the  lowest  task  and 
overall  performance  means. 

7,  Despite  the  fact  that  shift  operation  may  have  reduced  the  supervisors'  opportunity  to  observe 
incumbent  performance  below  that  of  peers,  sutx;rvisor  ratings  of  task  performance  appeared  to  be  shglitiy 
more  reliable  than  peer  ratings. 

8,  The  number  of  tasks  rated  prior  to  making  overall  performance  ratings  had  no  bearing  on  the 
correlation  between  task  and  performance  ratings.  AFSC  304X4,  with  far  the  most  tasks,  yielded  the 
highest  correlations  between  individual  task  ratings  and  overall  pcrfonnance  ratings. 

9,  Correlations  between  data  of  record,  such  as  grade  or  aptitude  test  scores,  and  task  pcrfonnancc 
ratings  tended  to  be  significantly  greater  than  ^iero.  Although  the  r's  were  smalU  they  were  systematically 
arrayed  instead  of  randomly  distributed. 


VI I L RFC  RTSS  ION  ANALYSES  OF  RFSTRICTFD  SAM  PLUS 

The  maximum  sample  analyses  made  up  to  this  point  have  been  representative  of  the  data  pool. 
However,  use  of  the  incumbent  records  with  incomplete  data,  such  as  test  results,  has  raised  questions 
regarding  sample  selection  for  future  analyses  to  be  made  on  variables  that  have  not  been  reported  here. 
Tlicse  variables  are  primarily  in  the  areas  of  vocational  interest  and  job  satisfaction.  Tlie  monitoring  agency. 
AFHRL,  has  performed  all  the  ensuing  regression  problems  in  order  to  select  a ‘‘flagged”  set  of  rating  data 
which  would  meet  requirements  for  future  analyses. 

The  ‘‘Flagged”  Sample  Concept 

A basic  population  of  criterion  ratings  was  so  chosen  that  any  new  set  of  analyses  would  involve 
identifiable  subsets  of  the  parent  population,  not  replacements  by  samples  of  unknown  representativeness. 
Tills  rule  will  hold  for  all  studies  in  which  a complete  set  of  criterion  ratings  is  required,  including  studies 
not  contained  in  the  present  report. 

Tlie  first  condition  for  selecting  a data  member  was  that  all  six  of  the  overall  performance  ratings 
were  present.  Supervisors  usually  assigned  all  six  ratings,  but  peers  often  skipped  the  first  (general 
performance)  altbougli  going  on  to  rate  on  the  five  subdimensions.  Such  a record  was  not  used,  and  another 
peer  or  supervisor  record  containing  full  data  was  chosen.  Tlie  goal  was  to  be  able  to  predict  every  kind  of 
performance  rating  for  the  entire  population.  Future  regression  problems  involving  new  predictor  variables 
would  necessarily  encounter  incumbents  with  missing  data  in  other  than  overall  performance  measures. 
Selecting  a flagged  population  makes  it  possible  to  rerun  any  of  the  original  analyses  on  the  subsamplc  to 
detemaine  if  selection  on  a new  variable  has  biased  the  sample.  Tlie  criteria  for  selecting  a record  to  be  used 
in  the  flagged  population  were: 

! . All  six  overall  performance  ratings  were  available; 

2.  performance  was  rated  on  at  least  one  task; 

3.  a combination  of  one  peer  and  one  supervisor  was  preferred,  but  not  required; 

4.  where  there  was  a choice  between  two  peers  or  two  supervisors,  the  rater  who  had  rated  on  more 
tasks  was  chosen; 

5.  not  more  than  two  records  per  incumbent  were  used; 

6.  in  the  absence  of  two  rating  records  one  record  was  used  if  it  was  complete.  (Tliis  was  most  often 
a record  Irotn  the  supplemental  survey,  which  employed  one  rater,  a supervisor.) 
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Cumbinitig  Raters 

A serious  dilTiculty  arises  when  many  of  the  ralccs  perform  different  tasks.  One  nuist  have  a true 
intercorrelation  matrix  to  compute  regression  problems:  a partial  overlap  will  not  suftlcc.  Nevcrtlieless,  it  is 
possible  to  predict  overall  performance  ratings  under  these  conditions  by  generating  a matrix  containing  a 
membership  variable  (dummy  variable)  for  each  task.  The  resulting  intercorrclation  matrix  contains  the 
criterion  vectors  (six  in  this  case)  and  2N  predictor  variables,  where  N is  the  number  ol  rated  tasks.  Tlie  task 
rating  data  ceils  of  the  compulation  matrix  contain  zeros  for  missing  data  and  numerical  values  where 
ratings  were  assigned.  Tire  dummy  vectors  contain  Is  where  lire  corresponding  rating  vectors  contain  Os  and 
Os  where  the  cor  responding  vectors  contain  ratings.  If  the  prediction  system  also  contains  other  types  ol 
data,  such  as  grade  and  months  of  service,  it  is  possible  to  handle  missing  data  in  the  same  manner. 
However,  it  was  not  necessary'  to  do  so  in  tliese  analyses  because  relatively  lew  incumbents  were  missing 
demographic  data  of  importance.  Loss  of  experimental  lest  battery  data  was  regarded  as  a basis  for 
dropping  the  case,  hut  regression  problcr^is  were  first  compirted  tor  the  entire  Haggcd  population  in  order  to 
dctcmiine  if  t lie  samples  which  had  been  retained  because  they  had  test  data  were  biased. 

Tlie  principle  of  one  incumbent,  one  criterion  rating,  was  applied.  If  there  were  two  criterion  raters, 
the  moan  rating  was  computed,  or  if  there  was  one  criterion  rater,  his  rating  was  used  with  weight  equal  to 
that  of  the  two.  Several  combinations  could  exist  for  the  task  performance  ratings,  liowever,  Aji  incumbent 
with  two  criterion  raters  might  be  rated  on  the  performance  of  a task  by  one  or  both  of  them.  It  was 
necessary  to  decide  whether  to  use  the  task  ratings  separately  or  to  compute  their  mean  as  a single 
predictor.  It  niiglu  be  argued  that  using  tlic  task  ratings  separately  would  affort  a more  defensible 
prediction  system,  since  it  takes  account  of  those  raters  wlio  tailed  to  observe  performance  on  a given  task. 
Regression  problems  were  run  with  tlie  six  criteria  for  eacli  AFSC  using  the  two  modes,  the  mean  task 
performance  rating  and  the  two  ratings  separately,  llie  resulting  R^s  were  very  close,  witl^  tlic  greatest 
observed  difference  .017.  In  tlie  18  possible  R^s  (six  criteria  for  each  of  tlircc  AFSCs)  15  computed  with 
mean  predictors  were  larger.  It  was  accordingly  decided  to  use  a one-man,  one-set  of  data,  analysis.  Tliis 
greatly  simplified  the  problem  when  data  of  record  were  introduced  into  the  prediction  system.  Tlie  R^s 
presented  in  Tabic  4,  and  following,  were  so  computed.  Table  4 is  arranged  in  three  major  columns  by 


Table  4.  Task  Performance  Ratings  Predicting  Overall  Performance 


Crnerlon 

AFSC  291X0 

AFSC  304X4 

AFSC  431  XIC 

r"' 

R 

r’ 

R 

R 

General  performance 

.2093 

.457 

.2905 

.539 

.2255 

.475 

Amount  of  work  performed 

.2241 

.473 

.3066 

.554 

.2089 

.457 

Quality  of  work 

.2321 

.482 

.2980 

.546 

.2042 

.452 

Will  exceed  his  share 

,2244 

.474 

.2935 

.542 

.2265 

.476 

Self-initiating 

.2172 

.466 

.2835 

.532 

.2242 

.474 

Will  share  his  knowledge  and  skill 

.2291 

.479 

.2802 

.529 

.2089 

.457 

Rating  Characteristics 

Number  of  tasks 

51 

95 

55 

Number  of  predictors 

102 

190 

1 10 

Number  of  incumbents 

1,254 

806 

1 J 

195 

1 
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AFSC.  The  actual  statistics  fall  under  the  headings  and  R.  The  last  three  rows  provide  the  number  of 
tasks  which  is  doubled  by  adding  the  dummy  variables  in  the  nc\t  row,  and  the  number  of  job  incumbents 
in  the  final  row.  Thus,  AFSC  291X0  had  51  rateable  tasks,  which  produced  a prediction  system  of  102 
predictors  when  accompanied  by  the  dummy  variables.  There  were  K254  incumbents  in  AFSC  291  XO,  and 
this  meant  that  each  correlation  coefficient  in  the  regression  equation  matrix  contained  1 ,254  observations. 
The  tasks  of  AFSC  291X0,  together  with  their  nonrated  membership  variables,  yielded  an  of  .2093  with 
the  criterion  of  general  overall  performance,  accounting  for  about  21  percent  of  the  criterion  rating 
variance.  Tine  table  contains  an  R^  range  from  .2042  ,3066,  and  AFSC  304X4  is  higliest  on  each  of  the 

six  criteria. 

Prediction  from  Data  of  Record 

Tables  5,  6,  and  7 give  the  zero -order  correlations  for  the  25  demographic  and  aptitude  variables  used 
in  subsequent  regression  problems  to  predict  the  six-criteria  overall  performance  ratings.  These  variables 
include  most  of  those  used  by  the  contractor  and  previously  identified  in  this  report,  Tlie  incumbent’s  age 
on  entering  the  Air  Force  has  been  employed  instead  of  his  date  of  entry  into  service.  Unfortunately  the 
variables  of  time  on  base  and  time  in  AFSC  had  to  be  drop|.:>ed.  Tliese  were  only  obtainable  from  job 
inventory  responses,  which  were  unavailable  for  ratccs  surveyed  in  the  supplementary  sample.  In 
subsequent  analyses  the  25  variables  have  been  classed  as  demographic  and  aptitude.  Tlie  aptitude  variables 
consist  of  II  experimental  tests  and  the  four  Aptitude  Indexes;  the  remaining  10  variables  are  the 
demographic  group. 


Tabic  5.  Correlations  with  Performance  Ratings  AFSC  291X0 

= 457 


Criterion^ 


Predictor  Variable 

Gerieral 

1 

Quantity 

2 

Quality 

3 

EKceeds 

Share 

4 

Self 

Motivating 

S 

Shares 

Knowledge 

6 

G radc 

105  H 

1449 

0969 

0864 

1029 

1068 

Total  active  federal  military  service,  TAI  MS 

0661 

0950 

0587 

0814 

0934 

0806 

Dccodint;  test  score 

0361 

0162 

0459 

{1249 

0381 

{1350 

Memory  for  landmarks  test  sane 

1008 

0688 

1 208 

0623 

0857 

0820 

Com  piex  scale  rcadinjt  test  score 

0200 

0099 

0222 

0357 

0268 

0024 

Pursuit  test  score 

0121 

{1476 

0423 

0282 

05  1 8 

0533 

1 igure  analogies  test  score 

0301 

0231 

0407 

0329 

0199 

0614 

Hands  test  score 

0985 

0840 

0673 

0823 

0728 

0479 

Cubes  test  score 

-03S5 

-0401 

0fH7 

-0057 

-Oil  1 

-0139 

Mecbanicai  principles  test  score 

0420 

-0524 

-0626 

0649 

-0623 

-0318 

I'ollowing  directions  test  score 

0664 

0098 

0560 

{1157 

0082 

0484 

Practical  estimations  test  score 

0460 

0323 

0725 

-0358 

0022 

0204 

Spatial  reasoning  tes!  score 

1053 

0630 

0994 

0649 

0450 

064(1 

Coded  1 if  married/  0 if  oiherwise  or  unknown 

0131 

0142 

0304 

{145{1 

-0426 

0226 

Coded  1 ii  divorced  or  separated/  U if  otherwise,  unk. 

0747 

0702 

0591 

0628 

041  1 

0591 

Code  1 il  single/  0 if  othei%vise  or  unknown 

0058 

-{1203 

0260 

0409 

{1444 

0201 

Size  of  city  of  origin 

-moi 

-0128 

0300 

0028 

-0183 

0145 

Mechanical  Aptitude  lndc\ 

-0112 

0317 

0219 

-0463 

-0077 

-0060 

Administrative  Aptitude  Index 

^0040 

-0269 

0036 

- 0226 

0321 

-0083 

General  Aptitude  Index 

i)078 

-0100 

0334 

-0180 

0274 

0006 

I'Icctronie  Aptitude  Index 

-0409 

-0683 

-0521 

-0451 

0624 

0162 

Coded  1 if  male/  0 if  femide  or  unknown 

0697 

-0623 

0946 

1 171 

0896 

-0773 

Coded  1 it  female/  0 if  male  or  unknown 

0697 

0623 

0946 

1171 

0896 

0773 

Age  at  time  of  enlistment 

0536 

0867 

0504 

0849 

0977 

0798 

I’dueation  level  at  time  of  enlistment 

0036 

0010 

0157 

0187 

0309 

0241 

potiUs  luivc  h\:cn  oniittotl 
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Table  6.  Correlations  with  Performance  Ratings  AFSC  304X4 

y = :m 


Criterion^ 


Predictor  Variable 


Exceeds  Self  Shares 

General  Quanlity  Quality  Share  Motivating  Knowledge 
{234  5 ^ 


Cirydc 

Totiil  active  federal  milUary  ser\  ice,  TAl  MS 

Decod  me  lesl  sc’t>re 

Memory  for  landmarks  test  score 

('o  1 11  p le  \ se  a I e re  ad  i ng  I e si  sco  r e 

Pursuit  test  score 

i i|iLire  amloeies  test  score 

I lands  test  score 

Cubes  lest  score 

Meclumicul  principles  tesi  score 

I’oUowinta  diredions  lest  score 

Priiclic;il  estimations  test  score 

Spilt iai  reusiiniri^  test  score 

Coded  1 il  married/  t)  if  otherwise  or  unknown 

Coded  I if  divorced  or  separaied/  0 it  olhcisvise,  unk. 

Coded  \ it  single/  0 if  oliierwise  or  unknown 

Size  of  city  of  origin 

Mechanical  Aptitude  Index 

Administrative  Aptitude  index 

(ieneral  Aptitude  Index 

Idcctronic  Aptitude  Index 

Coded  \ il  male/  0 if  female  or  unknown 

Coded  t if  female/  0 if  male  or  unknown 

Age  at  lime  of  enlisimenl 

i’.ducation  level  at  lime  of  enlisimenl 


1357 

HI  20 

0913 

0670 

0817 

0270 

0748 

0291 

1243 

0656 

0641 

0212 

0449 

-0503 

0435 

0161 

0187 

-01  10 

0362 

0304 

0075 

-0105 

0240 

0171 

0598 

0059 

0368 

0085 

tl603 

0590 

-0492 

-0205 

0240 

0329 

0461 

0365 

-0423 

^0679 

-0220 

- 0964 

-(^88 

-0815 

0681 

0494 

0237 

0026 

1220 

0793 

0912 

(1501 

0700 

0406 

06  K6 

0888 

1185 

1071 

(1544 

0399 

-0I  89 

0053 

0306 

(1838 

-0006 

0351 

0524 

0704 

0372 

- 0194 

-0161 

0202 

0637 

0388 

0205 

0067 

0456 

0531 

-0298 

0175 

-0041 

0142 

0103 

(1046 

-0274 

-0627 

0175 

-0598 

-0156 

-0267 

0743 

0399 

0352 

0240 

1041 

0830 

0300 

0412 

OHIO 

t)26l 

0530 

0583 

049H 

0360 

0285 

0022 

0104 

0162 

0496 

-0264 

0155 

0130 

-0600 

0549 

-0281 


0653 

0001 


1234 
0546 
0627 
0611 
1216 
0548 
0160 
0664 
0022 
081  1 
0347 
0053 
0710 
0292 
0304 
-0355 
0157 
0012 
-0375 
-0346 
-0046 


0213 

0088 


'’^Decimal  points  have  been  omitted. 


Table  7.  Correlations  with  Performance  Ratings  AFSC  431X1C 

,V  = 4(i7 


Predictor  Variable 

Criterion^ 

General 

1 

Quantity 

2 

Qua  lity 

3 

Exceeds 
Sli  are 

4 

Self 

Motivating 

6 

Shares 

Knowledge 

€ 

f itade 

2137 

1324 

1900 

1434 

1790 

1606 

Total  active  federal  mililary  service,  TAb’MS 

1338 

0798 

1290 

0818 

1053 

1324 

Decoding  test  score 

■U069 

0313 

-0460 

^0086 

-0443 

-oiai 

Memory  for  landmarks  test  score 

0898 

0778 

0492 

0957 

0840 

075 1 

Complex  scale  reading  lest  score 

-011  1 

-0187 

0121 

-0069 

-0137 

-0144 

Pur  sail  test  score 

-0288 

0438 

-0190 

-0419 

^0208 

-0275 

I’igure  analogies  test  SL\>re 

-0348 

-0455 

-0513 

-0313 

-0257 

-0148 

Hands  lest  score 

0745 

0708 

0757 

04 1 2 

0698 

0699 

Cubes  test  score 

-1)001 

-0108 

-0254 

- 0047 

-0191 

-0074 

Mechanical  principles  lest  score 

0438 

-0065 

0315 

0417 

0286 

0369 

1 ollovving  directions  lest  score 

0502 

1)329 

0499 

0378 

0420 

0286 

Practical  estimations  test  score 

(1734 

0414 

0589 

0551 

0483 

0576 

Spatial  reasoning  test  Scx>re 

0414 

0368 

06 1 3 

0459 

0332 

0029 

Coded  1 if  married/  0 if  otherwise  or  unknown 

1026 

0675 

0469 

0314 

0564 

0622 

Coded  1 if  divorced  or  separated/  t)  if  otheixvise,  unk. 

-0695 

^0608 

-0656 

-0344 

-0553 

-0201 

Code  1 if  single/  0 if  otherwise  or  unknown 

-0808 

-0445 

-0273 

-0130 

-0363 

-0545 

Size  of  city  of  origin 

008(1 

-0185 

-0289 

0318 

0447 

0294 

Mechanical  Aptitude  Index 

0612 

0900 

0706 

0796 

0669 

0498 

Administrative  Aptitude  Index 

(1184 

0244 

0349 

0362 

0293 

0298 

Ceneral  Aptitiulc  Index 

0232 

0599 

0319 

0568 

05  1 0 

024  1 

1 lectronic  Aplitudc  index 

0632 

0645 

0918 

0859 

0878 

0768 

Coded  1 if  male/  (1  if  female  or  unknown 

Coded  1 If  female/  l>  if  male  or  unknown 

- 

- 

- 

- 

Age  at  time  of  enlistment 

1245 

0727 

1378 

1102 

1293 

1532 

1 ducal  km  level  at  time  of  enlistment 

0589 

0594 

0637 

0335 

0444 

0579 

'*l)vciiiiiil  pitiiits  hiivf  amittL'd, 
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Comparison  R^s  are  presented  in  Table  H for  blocks  of  data  predicting  each  of  the  six  criteria  for  each 
AFSC.  Tlie  task  rating  row  entries  of  Table  can  be  compared  directly  to  the  R^s  of  Tabic  4.  Large 

Tabic  S.  Regression  Problems  Compared  for  Tlivee  AFSCs  - R^s 


Predictor  NuitiberoF  Predictors^  AFSC  — 


Crilcrian 

System^ 

291  XQ 

304X4 

431  XlC 

291  XO 

304X4 

431X  TC 

(iolierjl  raiing 

All  variables 

127 

215 

135 

.3402 

.3S37 

.3287 

I'ask  ratings 

102 

190 

I 10 

.2HI4 

.3319 

.27f^7 

Demographic 

10 

lu 

10 

.02K7 

.0227 

.f)568 

ApUtLide 

15 

15 

15 

,0463 

.0426 

3)272 

Amount  of  w ork  pcrl'ormod 

AH  variables 

127 

215 

135 

.3394 

.4449 

.328t) 

rating 

Task  ratin^is 

102 

190 

no 

.2915 

.3721 

.2995 

Demographic 

If) 

10 

10 

.03  OS 

.0133 

.02.^0 

Aptitude 

15 

15 

15 

.1l3bS 

.0261 

.0321 

of  work  r^iting 

All  variables 

127 

215 

135 

.3397 

.4126 

.3050 

Task  ratings 

102 

190 

1 10 

.2840 

.34H3 

.2403 

Demograpiuc 

to 

10 

10 

.0304 

.0187 

.0463 

Aptiludc 

15 

15 

15 

.0424 

.0306 

.04 1 3 

Will  work  more  llnm  rojuirvU 

All  v;iriablcs 

127 

215 

135 

.3494 

,4520 

.3241 

r filing 

Task  ratings 

l{i2 

190 

no 

.3024 

,4038 

J660 

Demograph  ie 

10 

10 

10 

.0424 

.0087 

.0293 

Apiiiude 

15 

15 

15 

.02S8 

.0421 

.0275 

Self-initialing  t needs  no 

All  variables 

127 

215 

135 

.3741 

.3892 

„3258 

pr<iddi[ig)  rating 

Task  ratings 

102 

190 

no 

.3304 

.3384 

.2686 

Dei  no  graph  ie 

10 

10 

10 

.t)34b 

.0128 

.0441 

Aptitude 

15 

15 

15 

,0269 

.0315 

.{1287 

Will  share  know  ledge  and  skill 

All  variables 

127 

215 

135 

.3666 

.3058 

,3015 

rating 

Task  ratings 

102 

190 

MO 

.3317 

.2821 

.2647 

Demographic 

10 

10 

10 

.0343 

.0210 

.0359 

Aptitude 

15 

15 

15 

.0184 

.0352 

.0223 

Note-  — Numbt-T  of  obscrvjtions  per  oocfbcicnt  - Eiumbt-T  <>f  intuoibcnt  r^tL-ts  AFSC  29  1 XO,  457 : AFSC  504X4, 
399^  AFSC  431  XIC,  4H7. 

of  j incTiiborsliip  dunitny  vjriablo  ( not  on  tjsb ) doubles  tUc  tnimbi^r  of  predictors. 

nc^Mgnjtrug  1 0 dcinngrLipblc  vari-iblcs  ii  iirbitr^iry  : tht'  itmIc/ foniLdi:  coding  results  in  one  iionrcduudLiul  VLirUbic  for 
AFSC  291  XO  land  no  new  variiblt:  for  AFSCs  304X4  jarid  43lXlCb  Probli^nis  vvert  computed  bcrforc  it  was  learned  that  full 
data  existed  for  dc'inograpbic  variables  and  that  the  uti known  coding  was  iiinpty. 


reductions  in  sample  Ns  have  taken  place  due  to  dropping  incumbents  willtoul  test  data;  AFSC  291X0  is 
down  from  1,254  to  457;  AFSC  304X4  down  from  806  to  399;  and  AFSC431X1C  down  from  I J95  to 
487.  Increases  in  have  occurred,  muglily  of  values  .07  or  ,08.  Tlicsc  are  appreciable  amounts.  Tlieirsize 
should  serve  to  caution  one  against  liberally  Interpreting  values  when  large  numbers  of  predictors  are 
used  with  relatively  small  Ns.  However,  comparison  of  the  values  in  Table  8 with  those  in  Table  4 reveals 
that  there  have  been  only  minor  changes  in  the  patterns  of  the  tluec  AFSCs.  Prediction  of  overall  ratings 
for  AFSC  304X4  remains  the  best,  and  AFSC  291X0  predictions  are  slightly  better  than  those  for  AFSC 
431X1C.  The  differences  between  the  latter  two  are  small.  Tlie  relative  order  of  prediction  among  the  six 
criteria  within  each  AFSC  is  not  entirely  consistent  between  the  two  tables.  Hov^'ever,  all  these  differences 
arc  trivial  compared  to  the  size  of  the  prediction  difference  between  task  prediction  and  that  available  from 
demographic  and  test  data.  Tire  R^s  from  these  data  of  record  are  about  oiie-tenth  the  values  of  the  task 
prediction  R^s. 


With  few  exceptions  the  R^s  of  Table  8 drop  in  direct  correspondence  to  tiic  number  of  predictors  in 
the  system.  It  might  be  concluded  that  predictiveness  of  task  ratings,  taken  item^by-item  is  not  greater  than 
that  of  the  data  of  record.  This  is  true  for  a single  task  when  zeros  have  been  introduced  into  the  rating 
vector  to  include  persons  who  were  not  rated  on  the  performance  of  that  task.  However,  by  consulting 
Figures  4 and  5 it  will  be  seen  tJiat  when  task  ratings  were  predicting  only  overall  perfomiance  of 
incumbents  who  performed  those  tasks,  the  correlations  ranged  from  .40  to  .60.  Sc[uared,  any  one  of  these 
vectors  could  account  for  from  16  to  30  percent  of  the  criterion  variance.  Table  8 consequently  presents  an 
ambiguous  situation  in  wlrich  the  higher  R^s  accompanying  task  ratings  could  be  attributable  eitlier  to 
numerous  predictors  or  to  substantial  relationships  between  task  performance  and  overall  performance 
ratings.  It  should  be  pointed  out  that  while  all  correlations  between  task  pertormance  ratings  and  overall 
ratings  are  positive  in  Figures  4 and  5,  in  the  matrices  involving  incumbents  who  did  not  perlorm  tasks 
there  arc  many  significant  negative  correlations  bet%veen  the  task  rating  vectors  and  the  criterion  vectors. 
Tims,  the  nature  of  the  buildup  of  is  a very  complex  interative  process  in  which  details  cannot  be 
determined  by  mere  comparison  of  values. 

Each  kind  of  predictor,  task  perfomiance  ratings,  demographic  data,  and  aptitude  measures  was 
combined  in  a single  equation  to  predict  each  of  the  six  criterion  dimensions  per  specialty  in  Table  8,  Hicn 
each  block  was  removed  to  leave  die  remaining  two  blocks  of  predictors  in  Tables  9,  10,  and  1 1.  All  full 
and  restricted  models  for  an  AFSC  appear  in  one  table.  F-tests  were  applied  to  determine  the  statistical 


Table  9.  Unique  Contributions  of  Blocks  of  Variables  to  Predicting 
Overall  Performance  Ratings,  AFSC  291X0 

of  observations  = 457;  number  of  Utsk  predictors  = 5 1 tmks  + 5t  dummy  membership 
variables  - 102;  number  of  demographic  mriables  = 10;  number  of  aptitude  measures  =15. 


Degrees  of  freedom^f*  by  problem;  task  removiid,  0/j  = 1 02^  Of 2 - 530;  demographic 
removed,  Dfi  =10,  Of 2 = 330;  aptitude  measures  removed,  Df^  = 1 5,  Df 2 = 330.^ 


Criterion 

Problem 

Full 

r’ 

Restricted 

Difference 

F -ratio 

Frobabillty 

Cicneral  performance  rating 

Remove  task  variables 

.3402 

3723 

.2679 

I3i33 

.039 

R e mo  ve  d emog  ra  ph  ics 

3402 

3194 

.0207 

1.0365 

-412 

Remove  aptitude  variables 

.3402 

3033 

.0368 

1.2282 

.248 

Amount  of  work  rating 

Remove  task  variables 

3394 

3656 

.2738 

1.3411 

,029 

Remove  demographics 

.3394 

.3160 

.0234 

F1701 

.310 

Remove  aptitude  variables 

.3394 

.3092 

.0302 

1.1)075 

.447 

Quality  of  work  rating 

Remove  task  variables 

3397 

3727 

,2670 

1.3085 

.041 

Remove  demographics 

.3397 

3153 

.0245 

1.2224 

,275 

Remove  aptitude  variables 

.3397 

3066 

.0331 

13  031 

.352 

Exceeds  share  rating 

Remove  task  variables 

.3494 

3734 

,2760 

1.3724 

.020 

Remove  demographics 

.3494 

3235 

,0259 

1.3158 

.220 

Remove  aptitude  variables 

3494 

3311 

.0184 

.6206 

.858 

Self-initiating  rating 

Remove  task  variables 

.3741 

.0606 

.3135 

F6207 

,001 

Remove  demographics 

.3741 

.3586 

.0155 

3167 

.613 

Remove  aptitude  variables 

3741 

3701 

,0040 

J412 

1,000 

Shares  knowledge  rating 

Remove  task  variables 

3666 

.0538 

.3128 

13978 

.001 

R c mo  VC  d c m ogra  p h ies 

3666 

3494 

.0172 

,897  1 

,536 

Remove  aptitude  variables 

3666 

3454 

.0212 

,7358 

.748 

^An  iirbitrary  decision  was  iiiadt?  in  the  case  of  tliL'  demographic  variables  to  use  10  the  base;  since  there  is  redund- 
ancy in  male/ female  and  married/separated/ single,  the  number  of  predictor  variables  has  been  overestimated^  and  upon 
correction  might  show  significance.  Earlier  problems  permitting  the  number  of  variables  actually  entering  the  equation  as 
base  were  discarded  because  they  were  suspected  of  overestimating  a signiftcances.  Recomputarion  can  be  made  with  9 
demographic  variable.s  for  AFSC  291X0,  and  B for  the  other  two  AFSCs  because  tliere  were  no  women. 

L 

^Sec  footnote  of  1 of  Table  11;  correct  to  S\  tasks. 
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Table  10.  Unique  Contributions  of  Blocks  of  Variables  to  Predicting  Overall 
Performance  Ratings,  AFSC  304X4^ 

-V«mtcro/otecn.<,/.om=  ,?09;  number  of  task  rating  pre, in  tors  - 95  + 95  , lummy  memlH^rship  variables  = 190- 
number  ofdemofrrapittc  predictors  = 10;  number  of  aptitude  predictors  =15. 

llegreesoffreedom  by  problem:  f>  tasks  removed.  Of,  = 190,  Of,  = 1 89;  demograpbie.t  removed, 

■Ji  - it/,  ~ aptitude  i^rkibtes  removed^  Df^  ^ 15,  /)/j  = fS9. 


Criterion 


Probletn 


Full 


General  perronnance  rating 
Amount  of  work  rating 
Quality  of  work  rating 
I xc  ecd  s jih  are  ra  t i ng 
Self-ini hating  rating 
Shares  knowledge  rating 


Remove  task  variables 
R e mo  ve  d em  ogra  p h i cs 
Remove  aptitude  variables 

Remove  task  variables 
Remove  demographies 
Remove  aptitude  variables 

Remove  task  variables 
Remove  demographics 
Remove  aptitude  variables 

Remove  task  variabies 
Re  m o ve  d e mogr a p Ji  res 

Remove  aptitude  variables 

Remove  task  variables 
Remove  demographies 
Re  m o ve  a pt  it  u d e var  i a b les 

Remove  task  variables 
R e in  o ve  d e mograp  li  ics 
Remove  aptitude  variables 


'See  footnoEc  I of  Table 


1 1 : correct  to  95  tasks. 
See  footnote  1 of  Table  9. 


Res^trlcted 


Difference 


F-ralio 


Frobabilhv 


.3837 

.054  3 

.3294 

.5490 

1 .00(1 

.3837 

.3659 

.0178 

.5465 

.855 

,3837 

.3486 

,0351 

.7177 

.765 

.4449 

.0393 

,4056 

,7506 

.974 

.4449 

.4294 

,0155 

,5271 

.870 

,4449 

.4233 

.0316 

.4913 

.943 

-4  1 26 

.0446 

,3680 

.6436 

.999 

.4126 

.3579 

,0547 

1.7612 

.070 

.4126 

.3889 

.0337 

.5094 

.934 

.4520 

.0489 

.4031 

.7555 

.972 

,4520 

.4361 

.0159 

.5484 

.854 

.4520 

.3956 

,0563 

1.2954 

.208 

.3892 

.0390 

.3503 

.5891 

1 .OtlO 

.3892 

.3779 

.0114 

,3521 

,965 

.3892 

.3795 

.0097 

.2010 

1 ,000 

.3058 

.0503 

.2555 

.3780 

1 .000 

.3058 

.2891 

.0167 

.4545 

.917 

.3058 

.2847 

.02  1 1 

.3826 

.982 

Table  1 1.  Unique  Contributions  of  Blocks  of  Variables  to  Predicting  Overall 
Performance  Ratings,  AFSC  431 XI C^ 

dumber  of  observations  = KIT;  number  of  task  rating  predictors  = .55  +55  duniniY  meniher.sbip  variabte.s  = 110; 
number  ofdemograpbiepredietors=  10;  number  of  aptitude  predictors  = 15. 

Degrees  of  fr.-edom  by  prohlemj*'  tasks  remoeeJ,  Df,  =110,  Df,  = 552;  (lemogruphie.s  remoeed. 

- 10,  D/j  = 352;  aptitude  variah leu  removed,  Of^  = 15,  Df^  = 552. 


Criterion 

Problem 

Full 

Rest  ricted 

Difference 

F-ratio 

Probability 

CUmeral  p<;rrormancc  rating 

Remove  task  variables 

.3287 

.0794 

.2493 

1 .1885 

. 1 23 

Remove  demographies 

.3287 

.2879 

.0408 

2.1391 

.02  1 

Remove  atptitude  variables 

.3287 

.306  1 

.0226 

.7898 

,689 

Amount  of  wcirk  rating 

Remove  task  variables 

.32811 

,0514 

.2766 

1 .3169 

-032 

Re  m 0 ve  d c m og  r a ph  i cs 

.3280 

.3152 

.0128 

.6689 

.753 

Remove  aptitude  variables 

.3280 

.3115 

.0164 

.5743 

.894 

Quality  of  work  rating 

Remove  task  variables 

.3050 

.0768 

.2281 

1.0503 

.365 

Remove  demographics 

.3050 

.2850 

.0200 

1.0132 

.432 

Remove  aptitude  variables 

.3050 

.2768 

.0281 

.9498 

.509 

Rxeccds  share  rat  ing 

Remove  task  variables 

.3241 

.0534 

.2707 

1 .28  1 4 

.048 

Remove  demographics 

.3241 

.2820 

,0421 

2.1905 

.01  8 

Remove  aptiludo  variables 

.3241 

,2852 

.0389 

1.3490 

,171 

Self-initiating  rating 

Remove  task  variable,^ 

.3258 

.0669 

,2589 

1,2288 

.083 

Remove  demographics 

.3258 

.2935 

.0323 

1.6863 

.082 

Remove  aptitude  variables 

.3258 

.2984 

.0274 

.9547 

.503 

Shares  knowledge  rating 

Remove  task  variables 

.3015 

.0543 

,2472 

1.1327 

.200 

Remove  demographics 

.3015 

.2738 

.0278 

1.3994 

.179 

Remove  aptitude  variables 

3015 

.2688 

.0327 

1 .0990 

.356 

i Ilk  luij  nioaei  contains  a Dlock  ot  task  ratings  an 
and  a block  tif  t 5 aptitude  measures:  restricted  models  were 
m odek 


d 55  dummy  variables, 
obtained  by  rein oving  on 


block  of  10  deinogn^pbic  variables, 
e of  tbe  three  blocks  from  the  full 


b,. 


See  footnote  1 of  Table  9. 
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significance  of  the  loss  due  to  removal  of  a block  of  predictors*  In  Tables  9,  ]0»  and  II  the  full  model 
equation  contains  all  predictors  and  the  restricted  model  contains  the  remaining  two  blocks  of  predictors* 
This  is  the  unique  contribution  of  one  kind  of  predictor  made  in  die  presence  of  die  other  two.  The 
significance  is  in  terms  of  probability,  the  number  of  times  in  1,000  trials  that  the  difference  in  R between 
the  full  and  restricted  models  was  attributable  to  chance. 

From  Table  9,  it  appears  that  the  contribution  of  task  ratings  to  die  prediction  of  AFSC  291X0 
performance  ratings  was  significant  statistically  for  all  six  criteria*  However,  the  number  of  predictors  is 
large  and  the  number  of  observations  per  correlation  coefficient  is  relatively  small,  which  is  a condition  that 
suggests  that  the  results  should  be  treated  cautiously*  Aptitude  data  did  not  make  a contribution  to 
prediction  in  the  presence  of  both  task  and  demographic  variables,  and  demographic  data  made  no 
contribution  in  the  presence  of  both  task  and  aptitude  data* 

From  Table  10,  it  is  plain  that  very  large  losses,  as  high  as  40  percent  of  the  criterion  accountability, 
are  nonsignificant*  No  difference  value  is  significant  in  Table  10.  The  reason  is  the  high  ratio  of  criterion 
predictors  (95  tasks,  or  190  predictors)  to  the  number  of  perfo nuance  observations  per  correlation 
coefficient* 

In  Table  11,  there  are  only  two  criteria  which  reflect  significant  unique  prediction  from  task  ratings 
for  AFSC  431X1C*  The  significance  is  marginal.  However,  in  this  AFSC  the  demographic  data  made 
significant  contributions  to  predicting  ratings  on  two  criteria,  general  performance  and  willingness  to  do 
more  than  one’s  share*  Aptitude  measures  made  no  significant  contributions  in  the  presence  of  both  task 
and  demographic  variables.  Again,  very  large  reductions  in  can  be  statistically  nonsignificant  when  the 
number  of  predictors  removed  from  the  system  is  high  in  relation  to  the  number  of  observations  per 
correlation  coefficient.  Wlien  tasks  were  removed,  a loss  of  25  percent  of  the  variance  prediction  was 
nonsignificant,  although  removal  of  the  block  of  demographic  variables  caused  the  loss  of  only  4 percent 
prediction,  wliich  was  significant* 

Reduction  of  Five  Task  Performance  Predictors 

All  in  all,  the  findings  from  inclusion  ol  all  task  ratings  are  suggestive  hypothesized  relationships,  but 
the  results  presented  in  Tables  9 through  1 1 are  not  clearly  delineated.  Tlie  obvious  solution  to  the  problem 
was  to  cut  down  the  number  of  task  rating  predictors.  Tlris  was  done,  and  the  results  appear  in  Table  12 
and  following* 

A sequential  set  of  five  tasks  was  chosen  in  advance  of  computing  regression  problems  by  dividing  the 
number  of  tasks  into  five  equal  intervals,  beginning  with  the  second  task  rated.  This  concentrated  tasks 
early  in  the  list,  which  was  intentional,  because  it  was  suspected  that  raters  may  have  given  tlieir  best  efforts 
during  tlic  first  period  of  the  survey*  Tltis  set  of  five  tasks  is  here  termed  the  sequential  set;  a second  set  of 
five  tasks  was  chosen  on  the  basis  of  their  probable  predictiveness,  and  these  are  termed  the  critical  set* 

Five  tasks  were  selected  in  order  to  make  the  number  of  predictors  the  same  as  that  of  the  smallest 
block  of  non  task  data,  the  set  of  10  demographic  variables.  Tliis  created  a new  problem  because  few,  if  any, 
tasks  in  AFSC  291X0  were  performed  by  as  many  as  one-half  of  the  incumbents*  In  AFSC  304X4  there 
were  some  tasks  on  wliich  less  than  10  percent  of  the  incumbents  were  rated.  Unless  an  effort  was  made  to 
find  five  popularly  rated  tasks,  it  would  be  possible  for  an  incumbent  to  enter  the  regression  problems  by 
being  a nonperformer  on  every  task*  Accordingly,  for  the  problems  using  carefully  selected  tasks,  two  bases 
of  selection  were  applied  jointly,  the  percent  of  incumbents  rated  on  the  task  and  the  point  of  entry  of  the 
task  rating  variable  into  the  iterations  of  a regression  pioblem.  Tlie  problem  used  for  that  purpose  was  the 
prediction  of  general  performance  from  all  variables*  appearing  in  the  first  line  of  Table  8.  In  general,  the 
membership  variable  entered  the  iterations  in  close  proximity  to  the  task  rating  variable,  but  not  always. 

Zero-order  correlations  are  given  for  both  sets  of  five  tasks  and  membership  variables  against  the 
criterion  of  general  overall  performance  in  Table  12.  Tlie  membership  variable  is  labeled  “Dummy.  Tlie 
intercorrclations  are  not  presented  in  Table  12;  they  were  quite  substantial,  ranging  from  *44  to  .89,  or 
from  -.44  to  -.89,  for  different  tasks  and  dummies*  Tlie  correlation  between  a task  rating  vector  and  its 
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Tabic  12.  Statistics  of  Selected  Tasks 


5 Predictive  Tasks  5 Sequence-Selected  Tasks 


Correlation  with  Correlation  with 

General  Performance  General  Performance 

List  % Rated  List  — ■ • ■ — — — — % Rated 

Number  Task  Dummy  on  Task  Number  Task  Dummy  on  Task 


AFSC  291X0 


02 

.0801 

-.0128 

31 

02 

.0801 

-0128 

31 

13 

-.0244 

.0685 

39  12 

-.0152 

0763 

36 

24 

.0089 

.0297 

26 

22 

.0040 

0387 

34 

40 

-.0695 

.1215 

24 

32 

.0209 

.0108 

24 

49 

-.0030 

.0470 

36 

42 

-.0387 

.1045 

24 

AFSC  304X4 

04 

.1274 

-.0722 

29 

02 

.1293 

-.0569 

37 

21 

-.0303 

.0709 

50 

20 

.0548 

.0138 

46 

67 

.0926 

-.0236 

50 

38 

.0173 

.0315 

72 

.0628 

.0152 

50 

56 

0417 

.0200 

^0 

88 

.0098 

.0504 

44 

68 

0796 

-.0134 

49 

AFSC 431X1C 

02 

.0299 

.0573 

51 

02 

0299 

.0573 

51 

04 

.1433 

-.0906 

33 

13 

.0580 

.0049 

43 

1 1 

.0196 

.0530 

52 

24 

.1060 

-0845 

27 

31 

.0356 

.0207 

48 

35 

.0469 

0070 

46 

40 

.0046 

.0455 

46 

46 

.0157 

.0427 

43 

duniiiiy  lay  between  -.94  and  " .995.  It  is  likely  that  clusters  of  tasks  corresponded  to  certain  jobs,  which 
helped  to  create  a correlation  that  depended  upon  the  fact  of  being  rated  on  a task  rather  than  upon  the 
perforniancc  rating  level.  Tliis  should  be  borne  in  mind  when  interpreting  these  analyses. 

The  sequential  sets  of  tasks  began  with  the  second  task.  When  the  critical  sets  were  chosen,  it  turned 
out  that  llic  second  task  was  one  of  the  best  predictoi’s,  which  resulted  in  its  appearing  in  both  the 
sequential  list  and  the  critical  list  for  two  of  the  AFSCs.  Otherwise,  the  sets  arc  not  duplicative.  Neither  set 
contains  any  of  the  very  rare  tasks,  and  it  seems  probable  that  most  of  the  incumlxMits  were  represented  by 
a task  performance  rating  at  least  once  in  each  set  of  computations.  This,  however,  is  a presumption,  and  it 
may  not  liold  true  for  AFSC  304X4,  which  had  95  tasks  lo  be  spread  over  399  ratces. 

Tables  13  througii  17  afford  comparisons  of  the  three  AFSCs.  Tlic  blocks  of  task  performance 
predictors  contain  10  predictors,  while  the  10  dcmograpliic  and  15  aptitude  variables  arc  unchanged  from 
the  previous  analyses.  Tlie  results  of  the  analyses  shown  in  Table  13  are  conclusive.  The  evidence  that  task 
performance  ratings  predicted  overall  pciformancc  variance  in  a way  that  aptitude  and/or  demographic  data 
could  not  is  inescapable,  in  all  problems  in  the  table  the  likelihood  that  any  difference  between  the  full  and 
restricted  models  could  have  occurred  by  chance  is  less  than  1 in  1 ,000  trials. 

Table  14  gives  the  unique  contribution  of  demographic  data  to  each  of  the  six  criteria,  and  Table  15 
does  the  same  for  aptitude  data.  !n  the  presence  of  task  ratings  combined  with  the  other  predictor  block 
neither  demographic  nor  aptitude  measures  contribute  much  toward  predicting  overall  performance 
variance,  generally.  Nevertheless,  there  arc  suggestions  of  unique  contribution  to  motivational  perforinance 
ratings  for  AFSC  291X0  from  demographic  data,  and  correspondingly,  contributions  to  general 
performance  ratings  for  AFSC  43  1 XIC.  No  statistically  significant  contributions  were  made  by  the  block  of 
aptitude  variables  reported  in  Table  15. 

Tables  16  and  17  were  developed  to  test  the  possibility  that  overlap  between  demographic  and 
aptitude  data  was  obscuring  the  contribution  of  some  variables  of  record  toward  predicting  overall 
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Table  13.  Unique  Prediction  Contribution  of  5 Task  Ratings  to  Overall  Performance^'  ^ 


Criterion 

Df 

AFSC  291X0 
^ = 10  Dfj  = 422 

AFSC  304X4 

Df,  = 10  Df^  = 364 

AFSC  431X1  C 

Of,  = 10  Df^  = 45Z 

Full 

Rest 

Diff 

F-rat. 

Prob. 

R^ 

Full 

Rest 

□Iff 

F-rat. 

Prob. 

R^ 

Full 

R^ 

Rest 

Oiff 

F-rat. 

Prob. 

5 Task5  Selected  for  Predictiveness  and  Percent  Performing 

General  performance  rating 

.2523 

.0723 

.1800 

10,1582 

.01)0 

.2313 

,0543 

,1770 

8,3799 

.000 

.2564 

.0794 

.1770 

10.7579 

,000 

Amount  of  work  rating 

.2458 

.0656 

.1812 

10.1502 

,000 

.1968 

.0393 

,1575 

7,1391 

.000 

,2435 

.0514 

.1921 

1 1.4772 

.000 

Quality  of  work  rating 

.2492 

.0727 

.1765 

9.9198 

.000 

.2164 

,0446 

,1718 

7,9820 

,000 

.2440 

,0768 

.1671 

9.9925 

.000 

Exceeds  share  rating 

.2547 

.0734 

.1812 

10,2605 

.000 

.2068 

.0489 

,1579 

7,2465 

.000 

.2320 

,0534 

.1786 

10.5104 

.000 

Self-initiating  rating 

.2759 

.0606 

.2153 

12,5479 

,000 

.1971 

.0390 

,1582 

7.1712 

.000 

.2298 

.0669 

.1628 

9.5559 

.000 

Shares  knowledge  rating 

.2311 

.0538 

.1772 

9,7365 

.000 

.1824 

,0503 

,1320 

5,8785 

.000 

.2055 

.0543 

.1512 

8,6024 

,000 

5 Tasks  Selected  By  Sequence 

General  performance  rating 

.2221 

.0723 

.1498 

8,1268 

.000 

.1980 

.0543 

.1438 

6.5346 

.000 

.2157 

,0794 

,1363 

7.8573 

.000 

Amount  of  work  rating 

.1989 

.0656 

.1332 

7.0184 

.000 

.1917 

.0393 

.1524 

6.8632 

,000 

,1888 

.0514 

.1374 

7 ,6563 

.000 

Quality  of  work  rating 

.2257 

.0727 

.1530 

8.3367 

.000 

.21  16 

.0446 

,1670 

7 .7084 

.000 

.2145 

.0768 

.1376 

7,9197 

,000 

Exceeds  share  rating 

.2072 

,0734 

,1337 

7.1170 

.000 

.2032 

.0489 

,1543 

7 .05  11 

.000 

.1897 

.0534 

,1363 

7.6013 

,000 

Self-initiating  rating 

J272 

.0606 

.1666 

9.0981 

.000 

.2024 

.0390 

.1634 

7.4586 

.(KIO 

.1815 

.0669 

.1146 

6.3283 

.000 

Shares  knowledge  rating 

.2062 

.0538 

.1524 

8.1027 

,000 

.1760 

.0503 

,1257 

5,5512 

.000 

.1711 

.0543 

.1 168 

6,3664 

.000 

^Thc  full  model  predictor  system  contains  5 task  ratings  and  5 dummy  variables,  10  demographic  variables,  and  15  aptitude  variables:  the  restricted  model  contains  10 
demographic  variables  and  15  tptltude  variables. 

^See  footnote  1 of  Table  9. 


Table  14.  Unique  Demographic  Contribution  to  Aptitude  and  5 Task  Overall  Performance  Predictions^  ^ 


Crjteri<xn 

AFSC  291X0 

Df^  =10  Df^  = 422 

AFSC  a04X4 

Df^  - 10  Df^  = 364 

AFSC  431  X1C 

Dfj  = 10  Df*  ^ 452 

Full 

Rett 

Diff 

F-rat. 

Profc. 

Full 

Rest 

Diff 

F-rat. 

Proto. 

FuM 

Rest 

Diff 

F-rat. 

Proto. 

5 Tasks  Selected  for  Predietiveness  and  Percent  Performing 

General  performance  rating 

J523 

.2286 

.0237 

1.3367 

.20S 

.2313 

.2025 

.0288 

1.3626 

.196 

.2564 

.22.37 

.0327 

1 .9868 

,033 

Amount  of  work  rating 

,3468 

.2238 

.0330 

1,2906 

.233 

.1968 

.1749 

,0219 

.9936 

.448 

.2435 

.2352 

.0082 

.4927 

,895 

Quality  of  work  rating 

.2492 

.2360 

.0132 

.7429 

.684 

.2164 

,1975 

.0189 

.8785 

.553 

.2440 

.2141 

.0299 

1.7879 

.060 

I'xceeds  share  rating 

.2547 

.2167 

.0379 

2.1487 

.020 

.2068 

.1903 

.0165 

.7570 

,670 

.2320 

.2158 

.0162 

.9549 

,482 

ScU -initiating  rating 

.2759 

*3331 

*0428 

2,4943 

.006 

.1971 

.1801 

,0170 

.7707 

*657 

.2298 

.2021 

.0277 

1 .6268 

,096 

Shares  knowledge  rating 

.2311 

.2037 

.0274 

1.5019 

.136 

.1824 

.1726 

.0098 

.4347 

,939 

.2055 

.1795 

.0260 

1.4805 

.144 

5 Tasks  Selected 

By  Sequence 

General  performance  rating 

.2221 

,2018 

.0203 

1.1007 

.360 

.1980 

4 796 

.0185 

.8387 

.591 

.2157 

.1862 

.0295 

1 .7007 

.078 

Amount  of  work  rating 

.1989 

.1881 

.0107 

.5653 

.842 

.1917 

.1775 

.0142 

.6411 

.778 

.1888 

.1768 

.0120 

.6688 

.754 

Quality  of  work  rating 

.2257 

.2042 

.0215 

1.1699 

.309 

.2116 

.2031 

.0084 

.3898 

.951 

.2145 

.1900 

.0245 

1 .4094 

,173 

Fxcecds  share  rating 

.2072 

.1741 

.0330 

1.7575 

.066 

.2032 

.1970 

.0062 

.2814 

.985 

.1897 

.1729 

.0168 

.9344 

.501 

Self-initiating  rating 

.2272 

.1945 

.0326 

1.7823 

.063 

.2024 

1956 

,0068 

.3121 

.978 

.1815 

.1562 

.0254 

1.4010 

.177 

Shares  knowledge  rating 

.2062 

.1887 

.0175 

.9308 

.505 

.1760 

.1658 

.0101 

.4483 

.932 

.1711 

.1429 

.0282 

1.5361 

,124 

full  model  predictor  system  contains  5 task  ratings  and  5 dummy  variables*  10  demographic  variables,  and  15  aptitude  variables:  the  restricted  model  contains  10 
task  variables  and  15  aptitude  variables, 

*^Soe  foottjote  1 of  Tabic  9, 


Table  L\  Unique  Aptitude  Contribution  to  Demographic  and  5 Task  Overall  Performance  Prediction^'  ^ 


Criterion 

AFSC  291X0 

Df,  = 1 5 Dfj  = 422 

AF5C  304X4 

Df j =15  Df  j = 354 

AFSC  431XIC 

Dfj  = 1 5 Df  j “ 452 

Full 

Rest 

Diff 

F-rat. 

Prob. 

Full 

Rest 

Diff 

F*rat. 

Prob, 

R^ 

Full 

R^ 

Rest 

Diff 

F-rat. 

Prob. 

5 Tasks  Selected  for  Predictive  ness  and  Percent  Performing 

General  performance  rating 

.2523 

.2153 

.0370 

1.3912 

.147 

.2313 

.1909 

.0404 

1,2742 

,216 

.2564 

.2361 

.0202 

.8194 

.656 

Amount  of  work  rating 

.2468 

,2200 

.0268 

1.0029 

.451 

.1968 

.1664 

.0305 

.9206 

,541 

.2435 

.2185 

.0250 

,9947 

.459 

Quality  of  work  rating 

.2492 

.2231 

.0261 

.9779 

.478 

.2164 

,1910 

.0255 

,7888 

,690 

.2440 

.2120 

.0320 

1.2764 

.213 

Exceeds  share  rating 

.2547 

,2350 

.0197 

.7434 

.740 

.2068 

.1688 

.0380 

1.1624 

.299 

.2320 

.2093 

.0227 

.8894 

.576 

S e ! f-  in  i tia  ti  n g rat  ing 

.2759 

,2600 

.0159 

.6178 

.861 

.1971 

.1693 

.0278 

.3400 

.633 

.2298 

.2025 

.0272 

1,0655 

.387 

Shares  knowledge  rating 

.2311 

.2193 

.0118 

.4307 

.970 

.1824 

.1543 

.0276 

,8177 

.658 

.2055 

.1806 

.0249 

.9461 

.512 

5 Tasks  Selected  By  Sequence 

General  performance  rating 

.2221 

.1843 

.0378 

1.3676 

.159 

.1980 

,1605 

.0375 

1.1358 

,322 

.2157 

.2005 

.0152 

.5824 

.889 

Amount  of  work  rating 

.1989 

.1797 

.0192 

.674S 

.810 

.1917 

.1639 

.0278 

.8347 

.639 

,1888 

.1645 

,0243 

.9041 

.560 

Quality  of  work  rating 

.2257 

.1872 

.0385 

1.3990 

.144 

.2116 

.1824 

.0292 

,8982 

.566 

,2145 

.1868 

.0277 

1.0635 

.389 

Exceeds  share  rating 

.2072 

.1838 

.0233 

.8282 

.646 

.2032 

.1696 

.0336 

1,0236 

.430 

.1897 

.1691 

,0206 

-7662 

.716 

Self-initiating  rating 

,2272 

.2094 

.0178 

.6478 

.835 

.2024 

.1798 

.0226 

.6867 

.798 

.1815 

.1561 

,0254 

.9347 

.525 

Shares  knowledge  rating 

.2062 

.1976 

.0086 

.3054 

.995 

.1760 

.1497 

.0263 

.7735 

.707 

.1711 

.1536 

.0174 

,6334 

.848 

tail  model  predictor  system  contains  5 task  ratings  and  5 dummy  variables,  10  demographic  variables,  and  15  aptitude  variables;  the  restricted  model  contains  10 
task  variables  and  10  demographic  variables, 
h 

^Sec  footnote  1 of  Table  9. 


Table  16.  Demographic  Contribution  to  Performance  Prediction  by  5 Tasks^'  ^ 


Criterion 

AFSC  29txo 

Of,  = 10  Dfj  = 437 

AFSC  304X4 

Of,  = 10  Dfj  = 379 

AFSC  431X1C 

Df,  = 10  Ofj  = 467 

Full 

Rest 

□Iff 

F-rat, 

Prob. 

Full 

Rest 

Diff 

F-rat. 

Prob. 

R= 

Full 

Rest 

Oiff 

F-rat. 

Prob. 

5 Tasks  Selected  for  Predictiveness  and  Percent  Performing 

General  performance  rating 

.2153 

J905 

.0249 

1.3843 

.185 

.1909 

,1566 

.0343 

1.6082 

.102 

.2361 

.2008 

,0354 

2,1613 

.019 

Amount  of  work  rating 

.2200 

.1926 

.0274 

1,5340 

.124 

.1664 

.1465 

,0199 

,9051 

,528 

,2185 

.2067 

,01 18 

,7070 

.718 

Quality  of  work  rating 

.2231 

.1891 

,0340 

1,9130 

.042 

.1910 

,1698 

,0211 

.9894 

,452 

.2120 

.1801 

,0319 

1.8908 

.044 

Exceeds  share  rating 

.2350 

.1957 

,0393 

3.2445 

.015 

.1688 

,1516 

,0171 

.7809 

,647 

,2093 

.1934 

,0169 

,9998 

.442 

Self-initiating  rating 

.2600 

.2229 

.0371 

2,1915 

.017 

.1693 

.1514 

.0180 

,8203 

,609 

.2025 

.1776 

,0249 

1.4578 

,152 

Shares  knowledge  rating 

.2193 

.1895 

,0298 

1.6661 

.086 

.1548 

,1362 

,0186 

.8342 

,596 

.1806 

.1546 

,0259 

1.4779 

,I44 

5 Tasks  Selected  By  Sequence 

General  performance  rating 

.1843 

,1632 

.0211 

1.1322 

.336 

.1605 

.1431 

,0174 

.7842 

.644 

.2005 

.1658 

,0347 

2.0290 

,029 

Ain o u n t 0 f WOT k rat  i ng 

.1797 

.1589 

.0208 

1.1077 

.355 

.1639 

.1510 

,0129 

.5845 

,827 

.1645 

.1493 

.0152 

.8498 

,581 

Quality  of  work  rating 

.1872 

,1609 

.0263 

1.4114 

.172 

,1824 

.1718 

,0106 

,4901 

.896 

.1868 

.1561 

.0306 

1.7598 

.066 

Exceeds  share  rating 

J838 

.1514 

,0325 

1.7381 

.070 

.1696 

.1601 

.0095 

,4329 

.930 

,1691 

.1639 

.0051 

.2890 

.984 

Self-initiating  rating 

.2094 

.1738 

.0356 

1 .9650 

.036 

.1798 

.1711 

,0087 

,4025 

.945 

.1561 

.1313 

.0249 

1.3768 

.188 

Shares  knowledge  rating 

.1976 

.1736 

,0240 

1,3070 

.224 

.1497 

,1329 

.0168 

.7503 

,677 

.1536 

.1231 

.0305 

1.6854 

,081 

^The  full  model  predictor  system  contains  5 task  ratings  and  5 dummy  variabics:  plus  10  demographic  measures,  the  restricted  model  contains  the  5 task  and  dummy 
variables, 

^See  footnote  1 of  Table 


Table  17.  Aptitude  Contribution  to  Performance  Prediction  by  5 Tasks^ 


AFSC  291X0  AFSC  304X4  AFSC431X1C 


Criterion 

Df  j 

= 15  Df 

; - 432 

Df, 

= 13  Df;  = 

= 374 

Df, 

15  Dfj 

= 462 

Full 

Rest 

Dilf 

F-rat. 

Prob, 

Full 

R^ 

Rest 

Oiff 

F-rat, 

Prob, 

Full 

Rest 

Diff 

F-rat, 

Prob, 

5 Tasks  Selected  fo?  Predictivefiess  and  Percent  Performing 

General  performance  rating 

,2286 

.1905 

.0381 

1.4243 

,132 

.2025 

,1566 

.0459 

1,4357 

,128 

2237 

,2008 

.0229 

.9080 

,555 

Amount  of  work  rating 

,2238 

.1926 

,0312 

IJ575 

.303 

.1749 

.1465 

.0285 

.8599 

.610 

.2352 

.2067 

,0286 

1.1501 

,309 

Quality  of  work  rating 

.2360 

.1891 

.0469 

1.7675 

,037 

.1975 

.1698 

,0277 

.8600 

,610 

,2141 

.1801 

.0340 

1,3334 

,178 

Exceeds  share  rating 

,2167 

.1957 

,0210 

,7735 

,707 

.1903 

,1516 

.0386 

1.1894 

,277 

,2158 

.1924 

.0234 

.9179 

,544 

Self-initiating  rating 

.2331 

.2229 

.0102 

.3835 

.983 

.1801 

.1514 

.0288 

.8750 

.593 

.2021 

.1776 

.0244 

.9421 

,517 

Shares  knowledge  rating 

.2037 

.1895 

.0142 

.5125 

.934 

.1726 

.1362 

.0364 

1.0966 

,358 

.1795 

,1546 

,0249 

,9329 

.527 

5 Tasks  Selected  By  Sequence 

General  performance  rating 

.2018 

.1632 

.0387 

1.3949 

.145 

.1796 

.1431 

.0364 

1.1070 

,348 

.1862 

,1658 

.0204 

,7714 

.710 

Amount  of  work  rating 

.1881 

.1589 

.0293 

1.0385 

,414 

.1775 

.1510 

.0265 

.8021 

.676 

.1768 

.1493 

.02  7 5 

1,0304 

,422 

Quality  of  w ork  rating 

.2042 

.1609 

,0433 

1.5668 

.079 

.2031 

.1718 

.0313 

.9798 

,476 

.1900 

.1561 

,0339 

1.2881 

.205 

Exceeds  share  rating 

.1741 

.1514 

.0228 

.7945 

.684 

.1970 

,1601 

,0369 

1.1469 

.312 

,1729 

,1639 

,0090 

.3349 

.992 

Self-initiating  rating 

.1945 

.1738 

,0207 

.7403 

,74  3 

.1956 

1711 

,0244 

.7576 

.725 

,1562 

.1313 

.0249 

.9087 

.554 

Shares  knowledge  rating 

.1887 

,1736 

.0151 

.5363 

.920 

,1658 

.1329 

.0329 

.9848 

.470 

.1429 

.1231 

,0198 

.7114 

,774 

^Thc  full  model  predictor  system  contains  5 tnsk  ratings  and  5 dummy  variables,  plus  15  aptitude  measures:  the  restricted  model  contains  the  5 task  and  dummy  variables. 


performance.  In  these  tables  the  other  set  of  variables  is  screened  out.  Table  16  reaffirms  the  findings  of 
Table  14  that  demographic  data  make  a contribution  to  predicting  motivational  rating  variance  for  AFSC 
291X0,  and  to  predicting  general  perfomiance  ratings  for  AFSC  431XIC.  Tfierc  is  a marginally  significant 
indication  that  quality  of  work  ratings  in  AFSC  431X1C  were  predicted  by  demographic  data*  Tlie  tasks 
selected  by  sequence  reflect  the  same  patterns  as  those  selected  for  their  probable  predictiveness,  but  with 
sliglitly  less  statistical  significance.  In  Table  17,  there  is  only  one  statistically  significant  aptitude  prediction, 
and  it  is  marginal.  However,  failure  of  a block  of  aptitude  predictors  to  make  a sigiiificanl  contribution  docs 
not  necessarily  mean  that  an  individual  experimental  test  would  also  fail  when  used  alone  with  task  ratings 
as  a baseline* 

Cross-Rater  Prediction  of  Overall  Perfonnance 

Could  rater  tendency  have  been  responsible  for  the  obtained  significances  of  contribution  of  task 
ratings  toward  predicting  overall  performance?  Tlie  same  individuals  rated  an  incumbenCs  task  and  overall 
performance,  and  averaging  two  raters  may  not  have  eliminated  the  rater  tendency  effect.  Tlie  solution  was 
to  compute  cross-rater  regression  problems,  which  meant  abandoning  the  supplementary  sample,  which  had 
only  one  rater.  The  resulting  Ns  were  drastically  cut,  eventuating  in  170  cases  for  AFSC  291X0,  148  for 
AFSC  304X4,  and  193  for  AFSC  43 1X1 C.  All  incumbents  had  task  ratings  from  both  raters,  but  not 
necessarily  tasks  in  the  two  lists  selected  for  regression  problems.  In  fact,  breaking  up  mean  scores  may  have 
created  a number  of  new  cases  with  no  rated  tasks*  Tlie  cross-rater  computations  shown  in  Table  18  employ 
the  task  performance  ratings  made  by  Rater  A to  predict  the  overall  performance  ratings  made  by  Reiter  B, 
and  vice  versa.  Tlie  significance  of  loss  on  removal  of  the  task  block  follows  the  format  of  Tabic  13.  The 
loss  significance  findings  negate  the  hypothesis  that  rater  tendency  was  responsible  for  the  contribution  of 
the  task  rating  variables  toward  predicting  overall  performance  ratings.  If  one  assumes  that  rater  tendencies 
are  uncorrelated  between  raters,  pronounced  rater  tendencies  would  lower  cross-correlations,  and  the 
correlations  might  be  lower  than  those  from  which  Table  13  was  computed*  In  fact,  the  correlations 
producing  Table  18  were  about  the  same,  and  the  R^s  of  Table  18  arc  considerably  higher  than  those 
shown  in  Table  13.  Tltis  is  probably  attributable  to  the  relative  size  of  the  number  of  predictor  variables  to 
the  reduced  size  of  tlie  number  of  observations  per  correlation  coefficient*  Table  18  is  presented  as  a test  of 
the  rater  tendency  hypothesis,  and  it  should  not  be  quoted  as  a reflection  of  the  entire  study,  which 
contains  an  additional  sample. 

Is  it  possible  that  the  predictions  found  here  are  attributable  to  the  incumbents  who  were  not  rated 
on  any  of  the  five  tasks  in  each  equation?  Tlicre  are  four  sets  to  be  considered  in  identifying  such  cases: 
Rater  A and  Rater  B,  each  rating  a set  of  five  critical  and  a set  of  five  sequential  tasks*  A count  of  the  cases 
with  zero  rating  data  was  made,  and  the  median  number  ol  each  of  the  four  conditions  was  found  for  each 
AFSC.  Converted  to  percentages,  the  medians  were  approximately  12  percent  for  AFSC  291X0,  5 percent 
lor  AFSC  304X4,  and  1 percent  for  AFSC  431X1C.  It  is  higlily  unlikely  that  incumbents  who  lacked  task 
ratings  altogether  could  have  greatly  inlluenced  the  results  for  AFSCs  304X4  and  431X1  C,  although  they 
may  have  had  some  effect  in  AFSC  291X0*  The  influence  of  the  large  group  of  incumbents  who  were 
nonrated  on  any  specific  task  is  a different  matter;  they  could  greatly  inniicncc  the  R^,  If  nonrated 
incumbents  had  higher  overall  performance  scores  than  their  accompanying  rated  incumbents,  the  result 
would  be  negative  Fs,  such  as  the  one  shown  for  task  40  under  AFSC  291X0  in  Table  12.  Here  the  task 
rating  predictor  correlated  -*07  with  the  criterion  and  the  dummy  correlated  + .12* 

In  sum,  the  tests  demonstrated  that  the  large  contribution  of  task  performance  ratings  toward 
predicting  overall  performance  was  not  attributable  to  rater  tendency*  Further,  a simple  count  showed  that 
it  is  highly  improbable  that  nonperformers  on  every  task  were  the  source  of  the  contribution  of  task  ratings 
to  R^*  One  attractive  explanation  for  the  findings  is  that  they  were  produced  by  a combination  of  rated 
performance  quality  on  difficult  tasks  and  ratings  at  the  top  of  the  scale  on  jobs  composed  of  simple  tasks. 

Contributions  of  Single  Variables  of  Data  of  Record 

Table  19  concludes  the  analyses  by  returning  to  the  data  on  which  Tables  13  througli  17  are  based*  In 
Tabic  19,  the  individual  contribution  of  each  of  the  25  predictor  variables  representing  data  which  were 
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Table  18.  Cross*Rater  Regression  Contributions  from  Sets  of  5 Task  Performance  Ratings 


AFSC  291X0 

N = 170  = lO^dfj  = 135 

AFSC  304X4 
N = 1 4®  df  j - tOi  df 

2=110 

AFSC  431X1C 

N = 19®  df  1 = lOj  df,  = 15S 

Criteria  n 
System^ 

Fult 

Model 

Rest. 

Model 

R“ 

Diff. 

R^ 

F- 

Ratio 

Prob.*> 

Full 

Model 

Rest. 

Model 

r’ 

Dlff. 

F- 

Ratio 

Prob^ 

Full 

Model 

Rest, 

Model 

□iff. 

F- 

Ratio 

1 . A o n B 

.2561 

.1034 

.1527 

2.771 

sm 

Tasks  Selected  for  Criticality 

.2933  .1246  .i687 

2.698 

.005 

.2291 

.1462 

.0829 

1.698 

.085 

B on  A 

.3599 

.2189 

.1409 

2.972 

•oo:*: 

.2632 

,1773 

.0859 

1.317 

,230c 

.2189 

,1278 

.0912 

1.844 

,057 

2 . Aon  B 

.2875 

.1591 

,1284 

2,433 

.on 

,2959 

.1389 

,1570 

2.519 

,009 

,2076 

.1009 

,1067 

2.127 

.025 

B on  A 

.2741 

,1807 

.0934 

1.736 

.079^ 

,2268 

,1181 

.1087 

1,588 

,119c 

.2478 

,1293 

,1186 

2.490 

,008*: 

3.  A on  B 

.2748 

.1328 

.1421 

2.645 

.006 

.2075 

,1118 

.0958 

1.3656 

.205 

.2102 

.1239 

.0863 

1.726 

.079 

B on  A 

.2311 

.1512 

,0799 

1 .404 

.185'^ 

.3000 

.2031 

.0968 

1,563 

.127 

.2545 

.1467 

.1078 

2.2850 

,016 

4.  A on  B 

.2832 

.1363 

,1469 

2.767 

.004 

.3367 

.1573 

,1794 

3.057 

.002 

.2515 

.1852 

.0664 

1.401 

.184 

B on  A 

.3029 

.1795 

,1234 

2.390 

.012^ 

.1948 

.1  1 15 

.0833 

1.169 

,319^' 

.2926 

.1786 

.1  140 

2.546 

.007' 

5 

' A on  B 

.2393 

.1693 

,0934 

1.736 

.079 

.3224 

,1403 

.1821 

3.036 

,002 

.2265 

,1604 

.0661 

1.350 

.209 

B on  A 

.2791 

.1820 

,0971 

1.818 

,063 

.2943 

.1674 

,1270 

2.033 

,036*: 

,2854 

.1610 

,1244 

2.750 

.004C 

6.  A on  B 

.2609 

.1220 

,1389 

2.538 

.008 

,2319 

,1342 

.0978 

1.438 

.173 

,2229 

.1209 

,1020 

2.073 

.030 

B on  A 

.2291 

.1795 

,0497 

0.8696 

,563^ 

.2183 

.1718 

.0465 

0.672 

,748 

.1573 

.1087 

.0486 

0.91 1 

.525 

1 . A on  B 

.2657 

.1034 

,1623 

2.984 

.002 

Sequence  Selected  Tasks 

.2775  .1246  .1528 

2.390 

.013 

.2086 

,1462 

.0624 

1.246 

.266 

B on  A 

23469 

.2189 

,1280 

2.646 

.2954 

.1773 

,1181 

1.895 

,053c 

,2650 

.1278 

.1373 

2.951 

.002C 

2.  A on  B 

.3014 

.1591 

,1424 

2.751 

.004 

.2994 

.1389 

.1605 

2.589 

.007 

.1922 

.1009 

,0913 

1.787 

.067 

B on  A 

.3675 

.1807 

,1867 

3,988 

.OOOC 

.2152 

.1181 

,0951 

1.365 

,206^ 

.2509 

.1293 

.1216 

2.565 

.007' 

3.  A on  B 

.2787 

J328 

,1460 

2.732 

.004 

,2601 

.1118 

,1483 

2.265 

,019 

.2704 

.1239 

.1465 

3.172 

.001 

B on  A 

.3825 

.1512 

.2313 

5.057 

,000*: 

.3119 

,2031 

.1088 

L787 

.071 

,2616 

.1467 

.1148 

2.457 

.009' 

4.  A on  B 

.2574 

.1363 

,121 1 

2.202 

.021 

.3267 

.1573 

.1694 

2.844 

.003 

,2783 

,1852 

,0931 

2.039 

.033 

B on  A 

,3592 

.1795 

,1797 

3.787 

.000*: 

,1720 

.1115 

,0605 

0.825 

.605*: 

.3109 

,1786 

1.323 

3.032 

.002' 

5.  A on  B 

.2868 

,1693 

,1175 

2.224 

.020 

,2872 

.1403 

.1469 

2.329 

,016 

.2734 

.1604 

.1  120 

2.432 

.010 

B on  A 

.2777 

.1820 

.0957 

1.789 

XI68 

,2509 

.1674 

.0335 

1,260 

.261 

.3025 

.1610 

.1415 

3.204 

.001' 

6.  A on  B 

.2630 

.1220 

.1410 

2,583 

.007 

,3195 

.1342 

,1853 

3,077 

,002 

.2130 

.1309 

,0921 

1.849 

.056 

B on  A 

.2992 

1795 

,1197 

2.306 

.016C 

.2312 

,1718 

,0594 

0.873 

.560*: 

.1742 

,1087 

.0654 

1.252 

.263 

^Numbers  1 -6  are  the  6 rating  dimensions  shown  in  Table  1 and  elsewhere.  A on  B means  task  performance  ratings  by  raters  A predicting  overall  racings  by  raters  B;  B 
on  A is  the  reverse.  The  full  model  contains  S task  rating  predictors,  5 dummy  predictors^  and  25  variables  of  record;  the  last  are  retained. 

^If  the  probability  value  for  either  member  of  a pair  is  not  greater  than  .014  the  cross-rater  prediction  can  be  considered  to  be  significant.  Pairings  failing  significance  are: 
2 for  AFSC  291X0;  3 for  AFSC  304X4;  4 for  AFSC  431 X 1C. 

‘"Considered  significant  as  a pair. 
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either  oq  record,  or  which  miglit  be  obtained  by  testing  the  incumbent,  is  examined  for  prediction  of  the 
six  overall  criteria.  All  three  AFSCs  are  presented  in  Table  19  \vith  the  criterion  variables  composing  the 
columns,  Tlie  values  shown  are  probability  estimates,  the  chances  in  1,000  that  the  obtained  difference 
between  the  full  and  restricted  models  could  have  arisen  by  chance*  All  values  exceeding  *049  have  been 
dropped,  leaving  only  prediction  contributions  that  are  significant  at  the  5 percent  level  of  confidence  or 
better*  Tire  full  model  contained  five  task  performance  ratings  and  their  five  dummy  variables,  plus  one  of 
the  predictor  variables  of  record.  Tlie  restricted  model  was  obtained  by  dropping  the  single  variable  of 
record  data*  Table  19  has  been  abridged  by  deleting  predictors  which  sliowed  no  significant  contribution 
for  any  AFSC.  This  resulted  in  dropping  the  scores  from  seven  experimental  tests:  decoding,  pursuit,  figure 
analogies,  cubes,  mechanical  principles,  following  directions,  and  practical  estimations*  Tlie  variable  relating 
to  size  of  the  city  of  origin  was  dropped.  Two  Aptitude  Indexes,  the  Administrative  and  Electronics,  both 
of  which  had  operated  to  select  a sample,  were  not  significantly  predictive,  and  these  were  deleted.  It 
should  be  borne  in  mind  that  the  samples  contained  399  observations  per  correlation  coefficient,  or  more, 
and  that  the  full  model  contained  only  11  predictors.  Consequently,  the  results  are  not  particularly 
impressive,  because  small  differences  in  prediction  can  be  statistically  significant.  The  spatial  reasoning  test 
score  appears  to  have  had  some  predictiveness  for  AFSC  291X0.  Also,  it  is  possible  that  the  women 
switchboard  operators  contributed  overall  performance  that  were  rated  higher  than  those  of  other  types  of 
personnel*  The  complex  scale  reading  test  seems  to  have  been  predictive  for  personnel  in  the  304X4 
specialty.  Also,  tlieir  status  as  divorced  or  separated  was  significant.  However,  this  latter  type  of  item 
depends  upon  very  few  cases,  and  the  relationship  could  easily  arise  from  chance  associations  due  to 
selecting  cases*  Grade,  experience,  age,  and  maturity  (all  closely  related  factors)  appear  to  have  been 
predictive  of  overall  performance  in  AFSC  43 IX  1C.  This  is  the  type  of  relationship  that  one  would  expect 
in  a specialty  where  most  of  the  incumbents  did  a majority  of  tire  tasks.  There  was  also  a marginal 
indication  that  tlie  memory  for  landmarks  test  was  predictive.  The  two  experimental  test  predictions  for 
AFSCs  291X0  and  304X4  are  provocative,  but  not  sufficiently  firm  to  be  a basis  for  recommending  a 
research  program* 


[X,  SUMMARY  OF  REGRESSION  PROBLEMS 

Tire  central  issue  of  the  regression  problems  was  to  determine  if  overall  perfonriance  ratings  could  be 
satisfactorily  predicted  from  data  of  record;  that  is,  whether  overall  ratings  could  be  predicted  from  such 
kinds  of  data  as  grade.  Aptitude  Indexes,  age,  education,  and  experimental  test  scores.  Tlie  question  was 
whether  the  individual  task  performance  ratings  predicted  more  of  the  overall  variance  than  could  be 
achieved  by  combinations  of  readily  obtainable  scores.  An  additional  question  concerned  the  specificity  of 
the  overall  performance  ratings  for  an  AFSC:  Was  the  composition  of  perfonnance  in  the  three  AFSCs 
discernibly  different?  The  within-rater  agreement  on  overall  ratings  set  an  upper  limit  upon  their 
predictability.  Roughly,  this  was  found  to  be  r - .80,  which  translates  into  a 64  percent  variance  limitation 
on  prediction.  AFHRL  obtained  (in  parallel  analyses)  cross-rater  predictions  that  would  translate  into  lower 
limits  of  prediction  at  20  to  25  percent  of  the  criterion  variance* 

In  order  to  utilize  the  samples  it  was  necessary  to  combine  data  from  incumbents  who  were  not  rated 
on  the  same  tasks.  Also,  in  order  to  answer  the  questions  about  the  relative  contribution  of  the  different 
kinds  of  data  it  was  required  that  there  be  data  of  record  on  all  ratees.  The  problem  was  approached  by 
developing  a ‘‘flagged”  population  whose  overall  criterion  scores  could  be  used  as  a standard  of 
predictability*  The  samples  taken  for  the  regression  equation  could  then  be  subsamples  of  the  flagged 
population,  and  their  representativeness  could  be  tested* 

The  requirement  for  experimental  test  data  narrowed  the  samples  by  at  least  half  the  cases.  At  the 
same  time  the  need  to  assemble  a true  intercorrelation  matrix  forced  the  use  of  membership  variables,  and 
this  doubled  the  number  of  predictors  for  task  rating  data*  This  simultaneous  doubling  of  the  number  of 
task  rating  predictors  and  halving  of  tlie  number  of  incumbents  brought  the  ratio  of  predictors  to 
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observations  per  correlation  coefficient  to  a barely  acceptable  minimum,  Aithougli  the  prediction  of  overall 
performance  variance  was  at  least  25  percent,  the  statistical  significance  was  marginal  because  of  the  large 
number  of  predictors. 

The  number  of  predictors  was  reduced  to  two  sets  of  five  task  performance  ratings,  together  with 
their  dummy  membership  variables.  Although  some  ratees  may  not  have  been  adequately  represented  by 
tadc  performance  scores,  the  method  provided  an  unqualified  determination  that  as  few  as  five  task 
performance  ratings  could  predict  20  percent  of  the  overaU  performance  variance.  This  prediction  was 
compared  to  a maximum  of  7 percent  overall  performance  prediction  from  demographic  or  aptitude 
data,  and  to  5 percent  unique  prediction.  That  is,  the  data  of  record  contained  only  a 5 percent 
determination  of  overall  performance  accountability  wliich  was  unavailable  from  task  performance  ratings. 
In  assessing  the  probability  that  task  ratings  were  making  a unique  contribution  to  predicting  overall 
performance  ratings  the  findings  were  virtual  certainty;  the  likelihood  that  the  differences  would  have 
arisen  by  chance  was  less  than  one  trial  in  1,000  for  every  dimension  of  all  three  AFSCs. 

Although  the  contribution  of  data  of  record  was  small,  a few  of  the  variables  made  significant 
increases  in  the  total  accountability  of  overall  performance  ratings.  As  one  might  expect,  grade  was  the 
largest  single  contributor,  accompanied  by  variables  with  which  grade  is  correlated.  Nevertheless,  certain 
specific  experimental  tests  were  predictive,  and  the  amount  and  significance  of  their  predictive  ness  differed 
among  the  specialties. 


X.  DISCUSSION  AND  CONCLUSIONS 

The  highest  correlations  on  task  performance  were  found  among  the  tasks  of  the  304X4  AFSC.  It  has 
been  found  possible  to  bring  some  information  collected  outside  this  study  to  bear  on  the  observation.  The 
comparative  task  difficulties  for  the  three  AFSCs  are  shown  in  Figure  7.  Based  on  a scale  of  7,  and 
reflecting  the  mean  of  NCOs’  ratings  of  the  length  of  time  it  would  take  to  learn  each  task  as  compared  to 
others  in  the  inventory,  the  distributions  in  Figure  7 show  that  the  tasks  of  AFSC  304X4  were  rated  as 
much  more  difficult  than  most  of  those  in  the  other  AFSCs,  About  one-third  of  the  tasks  used  in  the 
performance  study  had  to  be  omitted  from  this  graph  because  they  had  been  combined  in  a way  that  was 
not  the  same  as  the  inventory.  However,  it  was  ascertained  (by  examination  of  the  separate  difficulty 
ratings  of  the  tasks  that  the  contractor  had  combined)  that  had  the  omitted  tasks  been  included  as  given  in 
the  inventoiy  the  resulting  distribution  would  have  shown  at  least  as  great  a difference  from  the  others. 
When  rating  task  difficulty,  the  NCOs  were  restricted  to  the  inventory  presented  to  them,  and  the  data  of 
the  three  graphs  presented  in  Figure  7 have  no  common  standard  among  AFSCs.  Nevertheless  there  is 
reason  to  believe  that  a real  connection  exists  between  task  difficulty  and  the  rateability  of  task 
performance.  This  accords  with  the  presumption  that  in  order  to  rate  task  performance  reliably  there  must 
be  observable  differences  in  task  perfonnances.  It  follows  that  the  greatest  differences  in  task  performance 
should  occur  in  those  tasks  wliich  take  the  longest  to  learn.  Two  factors  govern  the  observed  data.  One  is 
that  a gradation  in  performance  must  be  present  if  one  is  to  see  that  a task  is  not  done  perfectly;  the  other 
factor  is  a sampling  consideration.  At  any  one  time  of  sampling  performance  there  should  be  individuals  at 
varying  stages  of  learning.  Neither  gradations  in  performance  nor  occurrence  of  individuals  at  varying  stages 
of  learning  is  to  be  found  for  simple  tasks  of  the  done/not  done  variety,  particularly  if  those  tasks  are 
sufficiently  simple  to  be  done  on  a single  verbal  instruction. 

About  20  percent  of  all  tasks  shown  in  Figure  7 lie  below  the  value  4 on  the  7-point  scale  of 
difficulty.  These  are  probably  the  type  of  task  that  is  eitlrer  done  or  not  done,  with  little  or  no  possible 
gradient  in  performance.  Efforts  are  being  made  to  investigate  the  nature  of  supervisor  agreement  on  the 
performance  of  jobs  with  a preponderance  of  such  tasks,  since  adaptability  to  jobs  of  that  kind  is  related  to 
job  satisfaction  and  may  throw  light  on  the  adaptability  of  certain  kinds  of  personnel  to  military  service. 
All  of  the  foregoing  emphasizes  the  desirability  of  discovering  predictors  of  performance  that  are  effective 
at  the  task  level.  Going  beyond  this,  the  observation  that  AFSCs  with  different  difficulty  levels  are  unequal 
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Figure  7,  Comparative  Task  Difficulty  Means  by  AFSC. 


in  performance  prediction  raises  the  question  of  differences  of  job  types  within  an  AFSC  Are  job  types 
unequal  in  performance  evaluation  when  unequal  in  difficulty?  Analyses  are  being  made  which  will 
determine  if  the  data  collected  in  the  present  study  are  sufficient  to  throw  light  on  this  additional  problem. 

One  must  look  with  care  at  all  predictors  in  the  present  study  because  the  three  AFSCs  have 
extremely  different  aptitude  cutoff  points  in  Air  Force  training  assignment.  Owing  to  the  high  correlation 
among  Aptitude  Indexes  the  three  samples  are  stratified,  and  tlris  is  apparent  on  all  experimental  test  score 
means  and  on  educational  background.  Despite  the  fact  that  the  members  of  the  304X4  AFSC  were  elite 
with  respect  to  aptitude,  their  overall  performance  rating  means  and  their  task  performance  rating  means 
were  lower  than  the  other  two  AFSCs.  Thus,  an  attempt  to  combine  the  three  samples  would  have  resulted 
in  mixing  incompatible  criteria  of  performance.  Not  only  were  the  performance  criteria  different  for  the 
three,  tire  distributions  of  their  predictor  scores  were  unlike.  Consequently,  the  AFSC  samples  must  be 
treated  as  populations,  and  generalizations  made  beyond  the  framework  of  a single  AFSC  must  be  very 
cautious.  Wtiilc  this  somewhat  limits  the  kinds  of  conclusions  one  might  like  to  draw  from  the  study,  it  is  in 
itself  an  observation  of  value  because  the  data  reflect  conditions  that  will  reappear  in  tlie  operational  Air 
Force. 

Small,  statistically  significant  contributions  toward  predicting  overall  performance  were  made  by 
specific  experimental  tests  for  some  specialties  but  not  others.  The  test  predictions  may  be  worth  further 
exploration,  but  on  a research  basis  to  discover  general  areas  of  behavior  that  are  not  tapped  by  existing 
batteries.  The  practical  effects  on  selection  and  assignment  would  be  slight.  Lest  one  become 
overenthusiastic,  it  should  be  remembered  that  a 2 percent  increase  in  overall  accountability  would  be 
statistically  significant. 

When  one  attempts  to  account  for  differences  among  the  regression  equation  patterns  predicting  the 
various  kinds  of  performance,  lie  is  faced  with  the  distinctions  in  working  conditions  afforded  by  the  tliree 
specialties.  For  example,  tire  Aircraft  Maintenance  Specialist  tended  to  have  jobs  which  included  a large 
number  of  the  listed  tasks  in  die  performance  survey,  lliis  was  quite  different  from  the  jobs  in  the  other 
two  surveyed  groups.  Possibly  the  fact  that  a single  Aircraft  Maintenance  Specialist  job  sampled  a large 
number  of  the  tasks  caused  a lack  of  specificity  among  the  predictors,  which  resulted  in  an  increase  in  the 
effect  ot  grade  and  length  of  service  on  predicting  overall  performance  ratings.  Very  discrete  jobs  marked 
the  Telecommunications  Operations  Specialist  assignment,  and  a wide  variety  of  jobs  could  be  found  for 
Ground  Radio  Communications  Equipment  Repairmen.  A continuous  flow  of  work  passes  through  a 
communications  center,  while  the  work  of  a repairman  is  marked  by  completion  of  tabor  on  a tangible 
piece  of  hardware.  Tlie  predictors  having  weight  for  overall  dimensions  may  offer  a clue  as  to  how  the  work 
situation  affects  the  human  requirements,  but  at  this  point  the  conclusions  would  be  largely  speculation. 
For  example,  die  dimension  of  quantity  of  work  was  better  accounted  for  in  the  304X4  AFSC  than  one 
might  expect;  but,  on  reflection,  it  seems  possible  that  quantity  of  work  in  this  case  is  a function  of  the  rate 
at  which  a piece  of  equipment  is  adjusted,  which  is  a matter  of  skill  rather  than  motivation  and  energy.  The 
results  of  this  portion  of  the  study  offer  attractive  leads,  but  they  are  not  definitive. 

The  enterprise  has  reatliiincd  tlie  thesis  that  the  assessment  of  job  performance  is  a complex  process. 
It  has  suggested  that  the  factors  involved  may  not  be  the  same  for  different  AFSCs.  One  of  the  strongest 
factors  which  distinguish  jobs  is  the  relative  difficulty  of  their  tasks.  More  difficult  tasks  were  associated 
with  more  measurable  performance  differences.  Tliis  was  evidenced  by  better  agreement  between  pairs  of 
raters,  less  piling  up  ol  ratings  at  the  top  of  the  scale,  and  better  agreement  between  task  performance 
ratings  and  ratings  ot  whole  job  pertorinance.  Nevertlieless,  die  performance  of  incumbents  in  jobs  made  up 
ol  less  discriminating  tasks  was  still  differentiated.  In  such  cases  it  seems  likely  tliat  other  aspects  of 
behavior  operated  to  distinguish  performance,  possibly  factors  of  motivation  and  attitude.  Demonstration 
of  that  hypothesis  must  await  future  analyses  of  these  data. 

From  its  outset,  the  study  was  known  to  be  a higli-risk  enterprise;  a serious  likelihood  existed  that  no 
definitive  findings  would  be  made.  Fortunately,  dearcut  observations  have  been  established.  Tliree  of  these 
appear  to  be  crucial:  (a>  perforniance  data  can  be  collected  at  the  level  of  the  task,  which  is  the  basic  unit 
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of  a job  inventory;  (b)  dirfcrences  in  task  performance  arc  observable  tor  those  tasks  fliat  NCOs  rale  as 
bein^  difficult  in  terms  of  required  learning  time;  and  (c)  excepting  behaviors  exhibiling  poor  motivation, 
persons  who  are  faced  wiili  rating  performance  of  very  simple  tasks  tend  to  credit  the  worker  with  perl  eel 
performance.  This  doesnh  seem  to  be  a great  deal  to  extract  from  such  a large  enterprise.  Could  not  almost 
anyone  have  reasoned  out  the  last  two  conclusions  without  tire  study  ? 

The  answer  is  yes,  maybe.  The  point  is  that  the  literature  on  job  performance  is  packed  with 
observations  about  inflation  of  ratings,  but  very  little  has  been  offered  which  associates  inllation  with  an 
honest  Inability  to  discriminate  between  performances.  Tlie  iinplicatinns  are  not  trivial.  Coupled  with  the 
finding  that  there  arc  tasks  whose  pcifonnances  can  be  discriminated,  it  is  a finding ol  immediate  value  to 
occupationaJ  analysis.  It  removes  a very  considerable  volume  of  tasks  Irom  consideration  in  terms  of 
aptitude  and  lengthy  training,  and  it  strengthens  the  need  to  concentrate  research  and  development  efforts 
on  tasks  that  have  been  selected  for  their  criticality. 

What  is  this  “criticality"’?  It  seems  to  be  a two-clement  concept.  Tlte  first  is  that  jobs  arc  critical  il 
failure  to  perform  them  properly  has  disastrous  results.  The  second  element  is  the  relative  dilllcuUy  of 
filling  the  jobs.  Since  all  jobs  contain  many  simple  tasks  as  well  as  complex  and  difficult  cues,  the 
composition  of  a critical  job  is  important.  First,  it  is  necessary  to  determine  which  tasks  make  the  job 
critical  in  the  sense  that  the  tasks  must  be  properly  done.  Next,  it  is  necessary  to  determine  il  the  critical 
tasks  arc  actually  difficult.  Finally,  if  the  objective  is  to  select  personnel  to  fill  those  jobs,  it  is  necessary  to 
find  an  adequate  sample  of  already  filled  jobs  to  use  as  performance  validators. 

The  last  requirement  is  often  very  difficult  to  meet.  Jobs  containing  the  critical  task  may  be.  and 
usually  arc.  in  small  numbers.  Incumbents  will  report  varying  proportions  of  their  lime  six^nt  on  the  tasks 
which  are  of  interest  for  training  and  assigning  personnel.  Consequently,  one  must  be  assured  in  picking  a 
sample  of  performance  validating  jobs  thal  he  is  avoiding  cases  that  were  overrated  because  the  incumbents 
spent  very  little  time  on  the  critical  tasks. 

The  findings  of  the  study  point  to  several  approaches  which  can  be  used  in  solving  the  validator 
problem.  Tlie  most  attractive  of  these  capitaliz.c  on  the  fact  that  task  difficulty  mcaiisiires  and  perfomiitnce 
discrimination  data  can  be  combined  with  job  clustering  techniques.  One  such  method  inigjit  be  to  combine 
job  inventory  responses  with  Automatic  Interaction  Detector  (AID)  (Koplyay,  Gott,&  Elton,  1973)  to 
detcmiine  tlie  relevance  of  certain  tasks,  or  task  clusters,  to  rated  overall  pertormaiice.  Tire  point  is  thal  we 
have  the  statistical  methods  at  hand  to  effect  improvements  in  identify ing  future  assignees  to  critical  jobs. 
Tire  results  of  this  study  arc  encouraging  in  that  they  indicate  the  feasibility  oi  obtaining  the  requisite  data. 

There  is  a grey  area  in  the  analyses  of  these  data  which  may  turn  out  to  be  important  for  practical 
purposes.  Statistically  significant  differences  were  found  in  performance  assessments  of  incumbents  with 
relatively  simple  tasks.  Wc  have  been  able  to  bring  to  bear  only  one  dimension  of  task  characteristics  as  a 
work  requirement.  NCOs’  task  difficulty  rating.  As  a result  of  this  limitation  we  have  assumed  that  some  of 
the  unmeasured  performance  variance  was  attributable  to  attitude  and  interest.  It  has  not  been  possible  in 
the  initial  analyses  to  break  this  category  down,  and  as  a result  the  explanation  may  be  an 
oversimplification.  One  can  cite  the  requirement  for  supervisors  to  instruct  subordinates,  and  tlie  need  to 
exercise  interpersonal  skills  in  the  process.  One  can  also  cite  the  possibility  of  lowered  pertormance  undci 
conditions  of  stress  and  emotional  tension,  which  might  affect  the  execution  of  very  simple  tasks.  This  is 
related  to  work  that  is  currently  going  forward  in  occupational  analysis  on  the  development  of  benchmark 
scales  for  several  dimensions  of  task  and  job  performance  (Goody,  1976). 


XS,  SUMMARY 


A study  was  organized  to  evaluate  the  potential  usefulness  of  rating  an  airman  s perl o nuance  on 
individual  tasks.  Broadly,  the  purpose  was  to  measure  job  performance,  but  more  particularly,  to  assess  the 
contribution  of  task  performance  ratings  to  such  measurement.  Making  assessments  at  the  task  level  has  tlie 
practical  advantage  of  using  that  unit  of  measure  men  I on  which  job  inventories  are  constructed  and 
analyzed. 
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The  study  was  necessarily  an  ambitious  one  because  of  the  inultitude  of  variables  that  might 
contribute  lo  performance  ratings,  whether  tficy  are  task  perfoniiance  ratings  or  overall  evaluations. 
Moreover,  the  study  had  to  be  representative  of  Air  Force  specialties  and  the  tasks  within  those  specialties. 
Willie  collecting  data  in  depth  it  was  reasonable  to  eollecl  corollary^  info  mi  at  ion  which  might  contribute  to 
understanding  the  hndings.  Tlius,  data  were  collected  on  sources  of  training  on  tasks,  on  the  motivational 
aspects  of  doing  tasks,  and  on  retention  of  skills.  The  underlying  aspects  of  evaluation  were  souglit,  both  at 
the  task  and  at  the  global  levels,  by  obtaining  self-ratings  on  task  performance  from  incumbents,  which 
were  paralleled  by  incumbent  assessments  of  the  tasks  as  motivators,  and  from  peers  and  supervisors,  who 
rated  task  performance,  task  performance  moderated  by  motivation  and  training,  and  six  kinds  of  overall 
pcrlormance*  Tlie  incumbent  data  bank  included  a current  job  inventory,  and  a lest  battery  containing  both 
experimental  cognitive  tests  and  measures  of  attitude,  job  satisfaction,  and  interests.  Air  Force  files  were 
mat  died  to  retrieve  such  information  as  grade,  education  at  entrance  to  service,  age  at  entrance  to  service, 
total  service  time,  and  Aptitude  Index  scores.  After  considerable  development  effort  utilizing  NCOs  as 
consultants,  the  following  specialties  were  chosen:  AFSC  29150,  Telecommunications  Operations 
Specialist;  AFSC  30454,  Ground  Radio  Communications  Equipment  Repairman;  and  AFSC  43151C, 
Aircralt  Maintenance  Specialist,  single- and  dual-engine  jet. 

Multiple  goals  were  listed  in  the  proposal  of  the  study,  and  in  recognition  of  the  immensity  of  the 
data  reservoir,  the  accumulated  information  was  also  regarded  as  a data  bank  for  exploration  of  promising 
leads,  and  for  aid  in  the  solution  of  continuing  methodological  problems,  A partial  list  of  the  goals  includes: 

1 Determining  ii  raters  can  agree  on  an  airman’s  quality  of  performance  on  individual  tasks; 

2.  Searching  for  aptitude  measures  not  in  current  use  which  are  predictive  of  task  or  job 
performance  ratings: 

3.  Evaluating  existing  aptitude  measures  as  correlates  of  performance; 

4.  Determining  which  items  of  record,  such  as  grade,  account  for  assessed  performance; 

5.  Assessing  motivation  and  measuring  its  contribution  to  performance  evaluations: 

6.  Assembling  numerous  measures  to  wciglit  their  relative  importance  in  accounting  for  work 
performance. 

Circumstances  at  the  time  of  the  study  threatened  to  reduce  the  number  of  cases  collected  below  the 
level  essential  to  studying  the  performance  of  individual  tasks.  A supplemental  survey  was  made  with  only 
one  supervisor  as  a rater  in  addition  to  the  incumbent.  Tliis  prevented  loss  of  the  critical  aspect  of  the 
study,  task  performance  assessment,  but  it  precluded  some  of  the  peripheral  analyses.  Among  the  casualties 
were  job  inventory  data,  which  contained  information  on  time  on  job,  and  a set  of  work  requirement 
ratings  applied  to  whole  jobs.  Air  Force  enlisted  discharges  proceeding  during  that  period  were  probably  the 
cause  of  reduced  returns  on  a Time  2 evaluation  obtained  from  supervisors.  Despite  all  of  this,  the  major 
purposes  of  the  study  were  met,  and  the  analyses  resulted  in  very  positive  rmdings. 

The  basic  findings  can  be  listed  as  follows: 

1 . Raters  can  agree  on  task  i^rformance  evaluations  to  a statistically  significant  degree. 

2.  inters  agree  better  on  rating  overall  Job  performance  than  on  rating  the  performance  of  single 

tasks. 

3.  Incumbents  arc  poorer  sources  of  task  ratings  than  peers  or  supervisors. 

4.  Peers  can  be  substituted  for  sut^ervisors  as  performance  evaluators  without  great  loss  in  reliability. 

5.  A few  task  ratings  taken  together  account  for  a substantial  percentage  of  the  overall  perfo nuance 
rating  variance. 

6.  Either,  or  both,  aptitude  data  and  demographic  data  (such  as  grade  and  length  of  service)  account 
ior  much  less  ol  the  overall  evaluation  of  an  airman’s  performance  than  can  be  had  from  ratings  on  as  few 
as  five  tasks. 

7.  It  is  likely  that  some  ol  the  unaccounted-for  overall  performance  variance  was  attributable  to  the 
attitudes  and  interests  of  the  incumbents.  (Only  preliminary  analyses  could  be  made  of  these  data,  but  the 
prediction  results  were  statistically  significant.) 
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8,  It  is  possible  that  small  iniprovements  can  be  made  in  selecting  ix^rsonncl  for  certain  task 
assignments  by  means  of  tests  not  in  current  use;  but  these  improvements  will  be  slight,  at  best. 

9.  Marked  differences  distinguished  the  304X4  AFSC  performance  ratings  from  the  other  two 
specialties;  they  were  internally  more  reliable  and  better  able  to  predict  overall  pcrfoimancc  ratings. 

10.  Use  of  the  top  of  the  rating  scale  was  frequent  when  performance  ratings  were  made  on  easier 
tasks,  and  on  performance  in  AFSCs  with  lower  aptitude  requirements. 

11.  Since  measurability  was  better  for  more  difflcnll  tasksj  with  less  use  of  the  lop  of  the  rating 
scale,  the  AFSC  with  high  aptitude  incumbents  received  the  lowest  mean  performance  ratings  of  the  three 
specialties. 

Tlic  importance  of  the  last  nnding  is  only  beginning  to  be  appreciated.  It  suggests  that  veiy  careful 
examination  should  be  made  ol  the  task  composition  of  jobs  in  tliosc  specialties  that  have  a substantial 
range  in  the  difficulty  of  tasks.  In  a sense,  performance  measurability  becomes  synonymous  with  task 
difllculty  as  it  is  being  operationally  recorded  for  Air  Force  job  inventorying.  This  provides  a preliminary 
index  of  tasks  that  can  be  used  to  identity  ditlicidt  jobs.  It  also  provides  a warning  that  job  pcrlormancc 
data  taken  without  regard  to  the  task  difficulty  composition  ol  jobs  used  as  validators  ot  selection 
procedures  can  yield  misleading  data.  It  would  seem  that  the  generality  ol  this  tmdiiig  ought  to  apply 
outside  the  Air  Force  as  well  as  in  it.  If  this  is  true,  it  is  likely  to  produce  reassessments  in  many  areas  ot 
validation  for  selection,  classillcation,  and  job  satisfaction  prediction. 
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NOTES  AND  STUDY  NUMBERS 


Work  was  performed  by  AFHRUs  Cooiputational  Sciences  Division  in  the  production  of  the  Contract 
Monitor’s  contribution  to  this  report.  Due  to  the  special  circumstances  under  wliich  work  has  been 
contmued  with  these  data  and  the  ongoing  nature  of  the  work,  it  is  thought  desirable  to  distinguish  the 
previous  analyses  from  any  that  may  have  been  commenced  concurrently,  but  which  have  not  been 
released. 

Computaitional  Sciences 


Division,  SM,  Study  Number 

Title  and  Comments 

6284 

Cross-Rater  Task  Prediction  of  Overall  Ratings,  TSKLVL,  consisting  in  the 
samples  reduced  to  incumbents  with  both  raters,  mostly  peer  and 
supervisor.  Tlie  two  sets  of  5 task  performance  ratings  by  one  category  of 
rater  were  used  in  a test  to  predict  tlie  overaD  ratings  made  by  the  other 
category  of  rater. 

5930 

Pay-off  Regressions  for  AIR  TSKLVL  Final  Report.  Regression  problems 
predicting  overall  criteria,  with  F-tests  by  blocks  of  predictors. 

5734 

Task  Performance  Predictions  of  TSKLVL  Overall  Criteria.  Inter- 
correlations  with  membership  variables  for  task  performance  ratings  with 
overall  criteria  using  “flagged”  samples. 

5687, 5668, 5530 

Title  not  relevant.  Tliese  were  preliminary  analyses  involving  regression 
prediction  of  overall  performance  from  cognitive  tests  and  arbitrarily 
weighted  noncognilivc  tests.  These  formed  the  basis  for  concluding  that 
attitude  data  would  contribute  to  predicting  overall  perfonnance, 
differentially  by  AFSCs.  Flagged  samples  had  not  been  developed,  and 
further  work  would  use  Gagged  samples  when  completed. 

5196 

Consolidate  F4 1609-7 1-C-OO 10  Survey  Data.  These  are  tlie  basic  record 
tapes  containing  the  major  protocols  and  the  test  data  for  the  entire  study. 
Matching  data  from  Air  Force  records  are  contained  under  other  study 
numbers. 

5135 

Sliakedown  Analysis  of  F41609-71-C-0010  Tapes.  Correlations  and 
intcrcorrelations  used  to  check  contractor’s  accuracy  and  quality,  along 
with  such  preliminary  analyses  as  seemed  useful. 
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